Understanding original file management

Joe-Engel wrote on 12/10/2021, 8:05 PM

I am new to MEPP 2022 and i need some help understanding best practices in regards to file and media management for MEPP.

I want to make our own home-movie style, “stories of our life” type projects, created from hundreds of photos and literally 10 terabytes of archived video footage. Each individual project might result in an hour of product on a USB thumb drive.

Each project might be culled from 2 terabytes of archived MP4s. These archives are often hours of dashcam footage from which multiple interesting segments, each of a few minutes, might be used.

My first concern is the huge file sizes. Does it make sense to drag 500GB files into a project just to access a few hundred MB for the end result? Won’t this just slow or choke everything? Would it make more sense use a third party to rough cut these files into small parts outside of MEPP?

Does MEPP have a very fast and easy to use “trimmer” function? If so can I use this to prune my clips, retain those, and release the archives files for future use?

Second concern is that the same archived file would be used in multiple projects. Maybe even the same footage. Would this confuse MEPP or corrupt the original archive?

I understand I have the option within MEPP, of pulling all of the original files used in a project into a new common location, but does this mean just the clips used or the full sized original files? It makes no sense to duplicate huge files in another location just for a few hundred MB. Remember I want to keep the master archive intact.

I hope all this makes some sense. I’m sure if I worked the application a while, I’d figure it out eventually. But it would be best to have a clear understanding before I end up with duplicates and disconnected source files across my many storage spaces.

Thanks

Comments

browj2 wrote on 12/10/2021, 11:11 PM

@Joe-Engel

Hi,

Good questions!

Does it make sense to drag 500GB files into a project just to access a few hundred MB for the end result? Won’t this just slow or choke everything?

I certainly hope that you don't have too many 500GB video clips. The biggest I have is 4.5GB as my cameras break up long shots into separate files.

Does MEPP have a very fast and easy to use “trimmer” function?

Yes. I do some rough editing/trimming at the source - preview (from the media pool using the brackets and then previewing in the Source Monitor), set range, import, preview next range, import,... until I have what I want from the file. On to the next file. See Multi-Movie below.

Note that upon importing the first clip, MEP creates 2 temporary files, .HDP which contains the small images that you see in the video object, and .H0 is the waveform data for the waveform display. Whether you import 10 seconds of a 1 hour clip or the full clip, it takes time for MEP to create the temporary files. Once done, the second and third...imports from the same file are fast. To answer another of your questions, if you import all or part of the same file in another project, MEP detects that the temporary files already exist and does not have to recreate them - it does not get confused.

...If so can I use this to prune my clips, retain those, and release the archives files for future use?

It doesn't work that way. MEP uses the files that are on your drive. It retains the path and filename and which part you want displayed on the timeline. That is all. The files always have to remain at the same location on your system or MEP won't find them the next time that you open a project. You need to watch my 2 tutorials on Basic Editing as this is fundamental to understanding how a video editing program works. The files are not locked, so they don't need to be released to be used in another or a thousand other projects.

The notion of "archive" is a backup somewhere on another drive, not the files that you would be using for editing. I trust that you have a backup of everything. A drive can fail; don't ask how I know.

Would it make more sense use a third party to rough cut these files into small parts outside of MEPP?

Not really as you have to review the file anyways, so you may as well do it once in MEP.

Second concern is that the same archived file would be used in multiple projects. Maybe even the same footage. Would this confuse MEPP or corrupt the original archive?

See above. No, MEP does not get confused and does not touch the files, it just uses them.

I understand I have the option within MEPP, of pulling all of the original files used in a project into a new common location, but does this mean just the clips used or the full sized original files? It makes no sense to duplicate huge files in another location just for a few hundred MB.

I don't use this. Some do so, because they archive (proper use of the term here) the project file with all of its material used. Thus, one would have a copy of the original material, archive it to a data disk or some other format (in duplicate) and then delete the material that was copied to the project folder. The original material never gets touched. This can be done at the beginning or end of a project.

Remember I want to keep the master archive intact.

Again, semantics, but the master is the original material, the archive is a copy archived somewhere else.

Multi-Movie

Whilst going through the material, if you see that you have material for more than one project, then open a second Movie (tab), change its name, and drag/import the pertinent clips (ranges) to that movie. I sometimes have 2 or movies going on in a project using material from the same source. Then each Movie can be exported and used as a separate Project. See my tutorials.

Time

Unless you are really doing nothing much in the way of editing, plan on a minimum of 1 hour per minute of finished product. If you get really good at creating videos, plan on much more time per hour.

Video Pro X (VPX)

Which is what I use. I depend on the Project Temp Folder in VPX for some file management and setting up for separate projects using parts of the same material, amongst other uses. Project Temp Folder (PTF) can be exported and imported into a new project. Nothing has to be on the timeline, just in the PTF.

VPX has 2 monitors, preview monitor for the timeline, and source monitor, which I find more convenient. MEP uses 1 monitor for both.

Hope I haven't confused you too much, or discouraged you.

John CB

browj2 wrote on 12/11/2021, 8:55 AM

@Joe-Engel

Hi Joe,

I just remembered a couple of other things.

These archives are often hours of dashcam footage from which multiple interesting segments, each of a few minutes, might be used.

One thing that I do with dashcam footage and other footage is to check the dialogue. Sometimes someone says something interesting or witty with lengthy periods of no dialogue. Thus, I look at the waveform to see if someone talks and listen to that. To do this, the clip has to be on the timeline and having audio and video on separate tracks simplifies the task. I always have AV on separate tracks. Set this up in the Program Settings and include show half waveform and always create waveform.

...home-movie style, “stories of our life” type projects, created from hundreds of photos and literally 10 terabytes of archived video footage.

You will want to complement this with other material, like scanned documents, Google maps and street images, Travel Maps, etc. I create a separate project folder in which I put all extraneous material, and a subfolder into which I'll save the exports.

There is much more to all of this and different ways to do the same thing.

John CB

Joe-Engel wrote on 12/11/2021, 3:55 PM

Hi John: Thanks so much for responding. You've given me much to think about. Some of the ideas you mention were bouncing around in my head and others I had not thought of. Some of what you wrote doesn't quite make sense to me so, if you have the patience, I would like to go back over to clarify a couple of things you mentioned. I will get back to this thread with more questions tomorrow, but one thing that might help is "what is the cache?"

Reading the manual, I don't see much explanation of cached files. Where they are? What they are? Where should they be? Do they get released or removed on app close. Can I delete?

What I called "archive" is a 15TB NAS. That's where I initially dump all of my footage from all of my devices. Videos from about 1980 to now. Also where I have all of my photos from over the years of our adventures. But I don't think working MEP projects directly from the NAS is a good idea, right?

Yes, actual files are 4GB chunks (or less), but to get a days worth of dash footage I might need 500GB worth of 4GB chunks. So I need to drag 12 (or more) of those 4GB files into a one day's project to see what I want to keep and then I might have to drag 5 days worth of chunks to create a whole story. That's a lot of files and a lot of video. My concern is do I have to retain all of that data someplace else on my faster storage, as I progress through my editing, even though I will ultimately be using actually maybe 5% or less of that.

So I also have a 3TB SATA and a 1TB SSD and a 1TB M2 "system drive" in my new I7-10400 desktop. I run the apps off of the system drive, but I try to keep data off of the system drive. Would the SSD be a good place for cache files?

Your audio tip about saving quips and comments is spot on! Those moments are priceless to us. So thanks for the waveform idea. That will make fast scanning through boring video to find those nuggets much easier.

Is there a "Okay I'm done, wrap it up" process? Where MEPP says it's does not need access to any of the large source files anymore and will then save JUST the parts used in the project? Is that what the render project does?

Finally... well finally for this post <G>. You reference your training. Where would I find that?

Thank you!

CubeAce wrote on 12/11/2021, 5:57 PM

@Joe-Engel @browj2

Hi Joe and welcome to the user to user forums.

When you say your files are all on an external NAS, how is it configured? Does it have built in redundancy? If a drive goes down can the missing data be reconstructed?

Cache files as far as I'm aware are always placed on the C: drive into a Temp folder but I could be wrong. I don't think there is a mention in the manual as to where they are put.

Ray.

Joe-Engel wrote on 12/11/2021, 7:12 PM

Hi CubeAce: Yes, the NAS is in its own RAID configuration and the data on the raid is further backed up to stand alone HD drives that are stored offsite. So that's pretty secure.

I still need to know where those cache files are. How big do they get? Etc, etc., etc.

browj2 wrote on 12/11/2021, 7:19 PM

@Joe-Engel

Hi Joe,

I am far from expert on this topic.

Definition (Wiki, of course) "In computing, a cache (/kæʃ/ (listen) kash,[1] or /ˈkeɪʃ/ kaysh in Australian English[2]) is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere."

This is something that I never think about and have not ever noticed. Undoubtedly, Magix is using caching so that future requests for the data can be served faster. The temp files that I mentioned is one of the methods that persists when the program is closed. When the program is open, then there is caching so that when you play back, you get a smoother playback (usually the second time round, hopefully). Magix uses rendered ranges as one of these methods. They are not persistent and are lost when the project is closed. You don't have to worry about them.

That said, your file system has to allow the creation of the temporary files in the folder that contains the files (video, audio, photo) that you use. So, your NAS is going to get a load of temporary files that can be deleted when done. If you delete any temporary files and then import the associated video file onto the timeline, the temporary files will be created again, slowing down the process, that is all.

My concern is do I have to retain all of that data someplace else on my faster storage, as I progress through my editing, even though I will ultimately be using actually maybe 5% or less of that.

Only if your NAS drive is extremely slow. It is not usually a problem. I have my video files spread over 3 hard drives, photos on 1. Just make sure that you have a backup. If you have a playback problem, it could be due to a number of factors other than the speed of the drive.

Speed usually becomes more important when you export. I export to an SSD drive that is there mainly for that purpose.

Is there a "Okay I'm done, wrap it up" process? Where MEPP says it's does not need access to any of the large source files anymore and will then save JUST the parts used in the project? Is that what the render project does?

Nope. I'm chuckling because the hardest thing for an artist is to say that "I'm done."

When you decide that you're done and render the project, either to a file like MP4 or DVD/BR, the program doesn't know or care. You decide when you're done. Then, you can archive the project, as I mentioned, by copying everything to a folder, and saving it all, MVP (project file) and all material used to an external storage medium. The thing is, if you have a 4.5GB video clip (15 minutes of FHD) and used only 10 seconds, the entire video clip is copied as that file is needed by the program if you ever open the project again. MEP does not create new files from what is only used on the timeline; it retains the information as to where the original files are located. You create the export or final product. Again, the project file in MEP does not contain any video, audio or photos at all, only the links to where they are.

Links to my tutorials on Basic Editing:

I suggest that you plan out a small project and start doing it. Then, you will find out more about how it all works.

I do test projects all of the time, just to play around with things. My tutorials and my test projects use pretty much the same video files, over and over. You can't break anything. g

We have only touched mainly on file management. When you get going, that is when you will start asking about the work flow.

John CB

Joe-Engel wrote on 12/11/2021, 9:03 PM

Hi John: On the subject of the NAS. My NAS is as fast as NAS can be, but the weakness of a NAS is it's connected via ethernet. 1 GB Ethernet is much slower I/O than internal SATA 6 HDD and a huge factor slower again than internal SSD and M2 PCIe storage. Further, I certainly do not want a bunch of temp or other extraneous files deposited on my NAS storage. That's to remain pristine as the archival repository for all our adventures.

So, it sounds like you're saying that for each project, I have to copy all the source files (500GB or more) to another internal drive storage location, start the project, ensuring that all the pertinent MEPP created working files also land on that internal drive, and build my project from there. I that correct?

Thanks for hanging in there with me on ths!

johnebaker wrote on 12/12/2021, 3:22 AM

@Joe-Engel

If you are worried about the amount of space the cache takes up - it is approx 30MB in size and you cannot alter its location.

The NAS box connection speeds compared to USB 3 and SATA are theoretically 1Gb/s, 4.6 Gb/s and 6Gb/s respectively.

In the real world these figures you get may be up to one half of the theoretical.

John EB

browj2 wrote on 12/12/2021, 9:04 AM

@Joe-Engel

Hi Joe,

Given the slow speed of the NAS and that you don't want temporary files on the NAS, then the easiest way is to check "Copy media to project folder" when you create the project. At the same time, give the project a name (not the date proposed by default) and check or change the Project folder.

As soon as you start importing from your NAS, you will get the following screen.

Select "Copy" and check the box "Don't show this message again. From then on, the files will be copied to your project folder and used from there. The temporary files will also be created there.

Of course, this reduces the efficiency of using the same material in more than one project because you will get a new copy each time and MEP will have to create the temporary files in the Project folder. I wouldn't worry about it.

John CB