Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically modify MediaSource ExternalTimedTextSources will not make subtitles effective #4899

Open
BigHeadDev opened this issue Nov 19, 2024 · 0 comments

Comments

@BigHeadDev
Copy link

BigHeadDev commented Nov 19, 2024

Describe the bug

I am using WinUI's Media related technology to create a subtitle player. My requirement is to create a subtitle editor. Here are my main processes:

  1. Select the file and load it into the player
  2. Use AI to generate subtitles and manually edit them
  3. Apply subtitles to the player and preview them
  4. Save subtitles to a file

However, I got stuck in step 3 and the generated subtitles cannot be updated instantly to the player. I cannot see any "subtitle" buttons in the player, so it seems that MediaPlayerElement does not support dynamic subtitle addition?

Steps to reproduce the bug

MainViewModel

[ObservableProperty]
private MediaSource media;

1,load a video file

[RelayCommand]
private async Task Import() {
    var openPicker = new FileOpenPicker();
    WinRT.Interop.InitializeWithWindow.Initialize(openPicker, WinRT.Interop.WindowNative.GetWindowHandle(App.Current.ServiceProvider.GetService<MainWindow>()));

    openPicker.FileTypeFilter.Add(".wmv");
    openPicker.FileTypeFilter.Add(".mp4");
    openPicker.FileTypeFilter.Add(".mpeg");
    openPicker.FileTypeFilter.Add(".3gp");
    openPicker.FileTypeFilter.Add(".avi");
    openPicker.FileTypeFilter.Add(".mkv");

    videoFile = await openPicker.PickSingleFileAsync();
    if (videoFile != null) {
        IsLoading = true;
        audioFile = await ExtractAudio(videoFile);
        
        Media = MediaSource.CreateFromStorageFile(videoFile);
        IsLoading = false;
    }
}

2,Binding in xaml

 <MediaPlayerElement x:Name="player"
                     AreTransportControlsEnabled="True"
                     Source="{Binding Media}">
     <MediaPlayerElement.TransportControls>
         <MediaTransportControls IsCompact="False" />
     </MediaPlayerElement.TransportControls>
 </MediaPlayerElement>

3,AI get subtitle

private async Task<TimedTextSource> ConvertToTimedTextSourceAsync(IEnumerable<Subtitle> subtitles) {
    string vttContent = "WEBVTT\r\n\r\n00:00:00.000 --> 00:00:05.120\r\n Today's video is about .net and specifically what is .net?\r\n\r\n00:00:05.120 --> 00:00:10.560\r\n What can you do with .net and some misconceptions that you may have about .net?\r\n\r\n00:00:10.560 --> 00:00:13.800\r\n Now, before we get started, I do want to announce my brand new course getting\r\n\r\n00:00:13.800 --> 00:00:15.320\r\n started with domain during design.\r\n\r\n00:00:15.320 --> 00:00:16.160\r\n Check it out.\r\n\r\n00:00:16.160 --> 00:00:19.600\r\n I think it's the most comprehensive course that you can find on domain\r\n\r\n00:00:19.600 --> 00:00:22.480\r\n drain design, taking you from the very basics assuming you have no\r\n\r\n00:00:22.480 --> 00:00:26.240\r\n knowledge in domain drain design and taking you up until some complex topics\r\n\r\n00:00:26.240 --> 00:00:28.440\r\n that you definitely need to know about.\r\n\r\n00:00:28.720 --> 00:00:32.960\r\n If you want to fully grasp and get the essence of domain driven design.\r\n\r\n00:00:32.960 --> 00:00:36.160\r\n So if that sounds intriguing, then make sure to check out the curriculum.\r\n\r\n00:00:36.160 --> 00:00:38.000\r\n The link is linked in the description.\r\n\r\n00:00:38.000 --> 00:00:41.920\r\n And if you're purchasing the course, then make sure to use the promo code M I D D\r\n\r\n00:00:41.920 --> 00:00:43.640\r\n D for a 20% discount.\r\n\r\n00:00:43.640 --> 00:00:45.080\r\n Now back to .net.\r\n\r\n00:00:45.080 --> 00:00:46.320\r\n .net is a framework.\r\n\r\n00:00:46.320 --> 00:00:50.560\r\n It's an ecosystem for developing applications initially in 2002.\r\n\r\n00:00:50.560 --> 00:00:55.320\r\n When .net was first released, then you could only use a Windows machine to\r\n\r\n00:00:55.320 --> 00:01:00.760\r\n develop .net applications. But that has changed dramatically over the years and\r\n\r\n00:01:00.760 --> 00:01:03.600\r\n specifically in the past 10 years.\r\n\r\n00:01:03.600 --> 00:01:07.400\r\n Net has made an incredible change that I think you should definitely know about.\r\n\r\n00:01:07.400 --> 00:01:11.640\r\n If you still think that .net developers are living in the 90s.\r\n\r\n00:01:11.640 --> 00:01:14.080\r\n So .net written by the cool kids.\r\n\r\n00:01:14.080 --> 00:01:16.440\r\n As you can see on the screen is first of all free.\r\n\r\n00:01:16.440 --> 00:01:20.080\r\n It's open source to confine all the source code of all the various frameworks.\r\n\r\n00:01:20.080 --> 00:01:22.080\r\n It's set run the languages on GitHub.\r\n\r\n00:01:22.080 --> 00:01:24.720\r\n It's open source and it's cross platform, which\r\n\r\n00:01:24.720 --> 00:01:29.920\r\n means you can develop .net applications on your Mac, your Windows or your Linux machine.\r\n\r\n00:01:29.920 --> 00:01:35.280\r\n But what is it actually when we say .net and we're saying it's a framework or an ecosystem,\r\n\r\n00:01:35.280 --> 00:01:38.720\r\n what does that include and what are .net developers?\r\n\r\n00:01:38.720 --> 00:01:42.360\r\n So when we say ecosystem, we're talking about the languages.\r\n\r\n00:01:42.360 --> 00:01:47.360\r\n So there's C sharp F sharp visual basic, all these various other languages under the umbrella\r\n\r\n00:01:47.360 --> 00:01:48.360\r\n of .net.\r\n\r\n00:01:48.360 --> 00:01:49.440\r\n We have the languages.\r\n\r\n00:01:49.440 --> 00:01:54.160\r\n We have a compiler that know how to take these languages and turn them into\r\n\r\n00:01:54.160 --> 00:02:01.840\r\n Microsoft intermediate language known as MSIL or just I L code similar to how Java and Kotlin\r\n\r\n00:02:01.840 --> 00:02:04.320\r\n are both compiled into byte code.\r\n\r\n00:02:04.320 --> 00:02:10.760\r\n So also .net is compiled into an intermediate language that is executed by some runtime.\r\n\r\n00:02:10.760 --> 00:02:12.560\r\n The runtimes are similar to your JVM.\r\n\r\n00:02:12.560 --> 00:02:17.720\r\n So it knows how to take the I L code and executed on various machines.\r\n\r\n00:02:17.720 --> 00:02:23.600\r\n That's why you can run your .net code on Windows on Mac on Linux on Raspberry Pis, et cetera, et cetera.\r\n\r\n00:02:23.600 --> 00:02:27.840\r\n Another important thing to note is the BCL or the base class library.\r\n\r\n00:02:27.840 --> 00:02:34.400\r\n This is a rich set of libraries that allow you to do almost anything you need from a mature language\r\n\r\n00:02:34.400 --> 00:02:37.400\r\n that you simply get out of the box and it evolves with time.\r\n\r\n00:02:37.400 --> 00:02:39.480\r\n And again, this also is open source.\r\n\r\n00:02:39.480 --> 00:02:42.080\r\n Now what can you actually build with .net?\r\n\r\n00:02:42.080 --> 00:02:47.640\r\n So everything that you see on the screen, you can develop with the .net framework.\r\n\r\n00:02:47.640 --> 00:02:52.680\r\n Now to get started with .net, all you need to do is go and download the .net.\r\n\r\n00:02:52.680 --> 00:02:57.360\r\n SDK. Once you've installed the SDK, then you can go ahead and use the .net CLI.\r\n\r\n00:02:57.360 --> 00:03:00.960\r\n Using the .net CLI, you can also create various applications.\r\n\r\n00:03:00.960 --> 00:03:06.600\r\n For example, this following line will create a concept application using the C sharp language\r\n\r\n00:03:06.600 --> 00:03:13.800\r\n and we'll put it in this folder and simply run .net run, which will run the default templates,\r\n\r\n00:03:13.800 --> 00:03:18.200\r\n concept application that will simply output Hello world to the screen.\r\n\r\n00:03:18.200 --> 00:03:21.760\r\n Now the beauty is that you can also do the following.\r\n\r\n00:03:21.960 --> 00:03:26.920\r\n You can say instead of C sharp, you can say F sharp, then over here, we can also say .net run.\r\n\r\n00:03:26.920 --> 00:03:34.240\r\n And this will run the F sharp template, even though these applications are written in different languages.\r\n\r\n00:03:34.240 --> 00:03:37.840\r\n As you can see, C sharp, F sharp and visual basic.\r\n\r\n00:03:37.840 --> 00:03:44.880\r\n If you're looking at how they're compiled, then you can see that all of them are compiled to a DLL file.\r\n\r\n00:03:44.880 --> 00:03:48.400\r\n This DLL file is the intermediate language that we talked about.\r\n\r\n00:03:48.600 --> 00:03:55.520\r\n Now, as you can see, the C sharp one is compiled to a DLL, but also F sharp and visual basic\r\n\r\n00:03:55.520 --> 00:03:58.880\r\n are also compiled into these DLLs.\r\n\r\n00:03:58.880 --> 00:04:06.400\r\n Now the beauty is that the runtime knows how to take these DLLs and compile them just in time\r\n\r\n00:04:06.400 --> 00:04:10.520\r\n into the native machine language needed for the machine that it's running on.\r\n\r\n00:04:10.520 --> 00:04:15.360\r\n And you have this interchangeability, even though they were written in completely different languages.\r\n\r\n00:04:15.400 --> 00:04:20.800\r\n Now we up until now looked at const applications and the simple example of a hello world,\r\n\r\n00:04:20.800 --> 00:04:25.040\r\n but you can build various other types of applications using .net.\r\n\r\n00:04:25.040 --> 00:04:26.800\r\n One of them is web APIs.\r\n\r\n00:04:26.800 --> 00:04:34.880\r\n So here's a three line example for building a web API that returns hello world from the default route.\r\n\r\n00:04:34.880 --> 00:04:37.360\r\n Then we have web applications for web applications.\r\n\r\n00:04:37.360 --> 00:04:38.680\r\n There are a few frameworks.\r\n\r\n00:04:38.680 --> 00:04:40.360\r\n One of them is Blazer.\r\n\r\n00:04:40.360 --> 00:04:44.760\r\n So here's an example of some code where you can see we have the data.\r\n\r\n00:04:44.760 --> 00:04:48.600\r\n The view part on the top and the code block in the bottom.\r\n\r\n00:04:48.600 --> 00:04:52.200\r\n This code, as you can notice, is simply C sharp.\r\n\r\n00:04:52.200 --> 00:04:57.760\r\n This means that you can use your favorite .net language for your front end as well.\r\n\r\n00:04:57.760 --> 00:05:02.880\r\n The reason why this is so powerful is because if you have a library that you like using,\r\n\r\n00:05:02.880 --> 00:05:05.600\r\n you can go ahead and use it from your front end, your back end.\r\n\r\n00:05:05.600 --> 00:05:09.400\r\n It's compatible across all the various .net applications.\r\n\r\n00:05:09.400 --> 00:05:10.760\r\n Then you have Nuget.\r\n\r\n00:05:10.800 --> 00:05:14.760\r\n Nuget is your NPM, your Mayvind Gradle PI.\r\n\r\n00:05:14.760 --> 00:05:18.560\r\n It's your package manager for .net packages.\r\n\r\n00:05:18.560 --> 00:05:23.920\r\n So if you're looking for a specific package, then you can go ahead search for it and find it over here,\r\n\r\n00:05:23.920 --> 00:05:28.520\r\n which means that you can go ahead and add any package that you want to your application,\r\n\r\n00:05:28.520 --> 00:05:29.520\r\n like so.\r\n\r\n00:05:29.520 --> 00:05:34.800\r\n The free open source ecosystem of .net packages is huge.\r\n\r\n00:05:34.800 --> 00:05:38.720\r\n And you can find a package for probably any need that you have.\r\n\r\n00:05:38.760 --> 00:05:39.920\r\n So that has been a taste of .net.\r\n\r\n00:05:39.920 --> 00:05:40.640\r\n I hope you enjoyed it.\r\n\r\n00:05:40.640 --> 00:05:45.480\r\n Any learn something new, make sure to smash the like button and smash the subscribe button if you haven't already.\r\n\r\n00:05:45.480 --> 00:05:47.160\r\n And I'll see you in the next one.\r\n";
    var stream = await ConvertStringToStreamAsync(vttContent);
    var timedTextSource = TimedTextSource.CreateFromStream(stream, "AI Subtitle");
    Media.ExternalTimedTextSources.Clear();
    Media.ExternalTimedTextSources.Add(timedTextSource);// NOT WORK
}

private async Task<IRandomAccessStream> ConvertStringToStreamAsync(string text) {
    var stream = new InMemoryRandomAccessStream();
    var writer = new DataWriter(stream.GetOutputStreamAt(0));
    writer.WriteString(text);
    await writer.StoreAsync();
    await writer.FlushAsync();
    stream.Seek(0);
    return stream;
}

Media.EternalTimedTextSources. Add (timedTextSource) failed to generate a subtitle button in MediaPlayerElement

Is there a way to insert subtitles into a media after opening it?

Expected behavior

  1. Load a file to mediasource
  2. Edit the subtitles
  3. Add subtitles to mediasource
  4. The subtitle button automatically appears in the UI and can be selected normally

Screenshots

1,Bug when insert subtitle after open media
Image

2,But, insert subtitle and then open media(I don't need this way)

Image

NuGet package version

Windows App SDK 1.6.3: 1.6.241114003

Packaging type

Packaged (MSIX)

Windows version

Windows 11 version 22H2 (22621, 2022 Update)

IDE

Visual Studio 2022

Additional context

waiting for your anserwer thanks~~~~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant