Minimising project media backup storage

terrypin wrote on 7/7/2017, 8:24 AM

What do others do to keep the disk storage space needed for project and media backups to reasonable levels when the project contains one or more fragments from an earlier finished video or other very large file please?

Suppose a 100 MB project includes a 10 MB extract made by trimming a 5 GB MP4, typically the finished video from a previous project. The routine backup using  File > Backup copy > Copy project and media into a folder will include the entire 5 GB file, the great bulk of it being redundant.

Some 10 or so of my 'daily sub-projects' in my current fortnight's holiday project may need extracts like that, from largely different large MP4 or AVI files. So, if I b/u all of them individually, I could waste 50 GB.

My current intention therefore is to do the trimming separately (in a 'trimming project'), export the fragments, and re-import those as needed into the holiday projects/sub-projects.

Anyone have a simpler method please?

--------------------

More generally, do others in fact always backup all projects? Until quite recently, I haven't done so - but occasionally regretted it.

Terry, East Grinstead, UK

 

 

Terry, East Grinstead, UK. PC: i7 6700K, 4.0 GHz, 32GB with Win 10 pro. Used many earlier versions of MEPP, currently mainly MEPP 2016 & 2017 (Using scores of macro scripts to add functionality, tailored to these versions.)

Comments

johnebaker wrote on 7/7/2017, 1:25 PM

Hi Terry

. . . . Anyone have a simpler method please? . . . .

I do not use the MEP backup functions for backing up my projects.

I have recently installed a 4th hard drive specifically for backing up the current working drive after having some issues - looked like the HD was beginning to fail, however it is OK and passes all the diagnostics.

Once a project is complete - I keep each project in its own folder, with the exception of what MEP creates in the AudioTemp folder - I use SyncBack to differentially backup the root folder containing all my projects folders which contain the original source video, images, audio etc and the AudioTemp folder to pairs of external hard drives. All temporary files created by MEP ie the .HDP, .H0 and BAK files are excluded from the backup.

Each pair of drives contain the same data, ie there is a backup of the backup drive, in case of failure of one, which I have had before.

Once a pair of drives are full I add another new pair, I have been using this method for over 10 years and is cost effective as storage is relative cheap - I am now on my 4th pair of external drives.

I did considered consolidating all the drives into a NAS storage box, however the economics of it is not cost effective and is 'putting all your eggs in one basket'.

I did consider zipping up the projects, however the saving on space is not significant enough , ~ 5%, to warrant this.

HTH

John EB

 

VPX 16, Movie Studio 2025, and earlier versions 2015 and 2016, Music Maker Premium 2024.

PC - running Windows 11 23H2 Professional on Intel i7-8700K 3.2 GHz, 16GB RAM, RTX 2060 6GB 192-bit GDDR6, 1 x 1Tb Sabrent NVME SSD (OS and programs), 2 x 4TB (Data) internal HDD + 1TB internal SSD (Work disc), + 6 ext backup HDDs.

Laptop - Lenovo Legion 5i Phantom - running Windows 11 24H2 on Intel Core i7-10750H, 16GB DDR4-SDRAM, 512GB SSD, 43.9 cm screen Full HD 1920 x 1080, Intel UHD 630 iGPU and NVIDIA GeForce RTX 2060 (6GB GDDR6)

Sony FDR-AX53e Video camera, DJI Osmo Action 3 and Sony HDR-AS30V Sports cams.

terrypin wrote on 7/7/2017, 2:50 PM

Hi John,

Thanks for that detailed description of your backup system and workflow. A very different method to mine, but I see its advantages.

Terry

Terry, East Grinstead, UK. PC: i7 6700K, 4.0 GHz, 32GB with Win 10 pro. Used many earlier versions of MEPP, currently mainly MEPP 2016 & 2017 (Using scores of macro scripts to add functionality, tailored to these versions.)

browj2 wrote on 7/7/2017, 10:17 PM

Hi, I wrote a long message, using my tablet. I never get logged out but this time I did. I did do a copy to clipboard before saving the comment but it was gone when I logged in. Oh well.

We're out camping again. Going fishing all day tomorrow.

Regards,

John CB

John C.B.

VideoPro X(17U); Movie Studio 2025 Platinum; Music Maker 2025 Premium Edition; Samplitude Pro X8 Suite; see About me for more.

Desktop System - Windows 11 Pro 24H2; MB ROG STRIX B560-A Gaming WiFi; Graphics Card Zotac Gaming NVIDIA GeForce RTX-3060, PS; Power supply EVGA 750W; Intel Core i7-10700K @ 3.80GHz (UHD Graphics 630); RAM 32 GB; OS on Kingston SSD 1TB; secondary WD 2TB; others 1.5TB, 3TB, 500GB, 4TB, 5TB, 6TB, 8TB, 12TB, 14TB; three monitors - HP 25" main, LG 4K 27" second, HP 27" third; Casio WK-225 piano keyboard; M-Audio M-Track USB mixer.

Notebook - Microsoft Surface Pro 4, i5-6300U, 8 GB RAM, 256 SSD, W10 Pro 20H2.

YouTube Channel: @JCBrownVideos

terrypin wrote on 7/8/2017, 2:11 AM

Hi John,

Been there...greatly frustrating. With longish compositions I do as you did, in my case with a hot key macro, pasting to a text file. Still need to assemble screenshots etc again though.

Ah, fishing - an obsession in my 20s/30s. Still have much of my gear. Passing rivers or lakes ** during country walks inevitably stirs longings and half-hearted vows to get into it again, but I never do....

** I was going to abbreviate those two nouns to the single 'water', but happily thought better of it!

Terry, East Grinstead, UK

Terry, East Grinstead, UK. PC: i7 6700K, 4.0 GHz, 32GB with Win 10 pro. Used many earlier versions of MEPP, currently mainly MEPP 2016 & 2017 (Using scores of macro scripts to add functionality, tailored to these versions.)

Recycler wrote on 7/11/2017, 12:33 PM

Terry:

FWIW, here is my arrangement. I worked with other video editors before coming to Magix, so my project folder arrangement is historic and does not follow any of the Magix assumptions!

Each new project begins with me copying and pasting a copy of my template of folders into the work area (see diagram). The first thing I do is to rename the top level folder to something unambiguous to identify the project uniquely (see later). The sources are brought into the appropriate folders as required.  The DTP folder is for the disc label print and jewel case insert work.

When editing begins, the project files go in their sub-folder.

So throughout work on the project everything I need is in that pack of folders. The most demanding of the various work that I do is recording concerts in 1080p/50, and with four cameras involved a project can easily be over 100 GB.

I run "BvckUp2" on a periodic (6 hour) basis to keep a working copy on a RAIDed NAS. [https://www.bvckup2.com, an elegantly coded software which runs efficiently with minimum impact and is highly recommended]. Call me paranoid but every fortnight I also copy all work to a disconnected non-RAID disc as an anti-ransomware protection (via  bvckup2 again).

Clearly as an amateur I cannot afford to carry this sort of data burden for ever, so once I am satisfied with the project, I preserve only a set of tracked WAVs for the CD, an ISO for the DVD version, and these days a decently high bit-rate H.265 version. Then I take a deep breath and dump everything else, except the DTP necessary to create more discs if required.  The slimmed down project remnant (perhaps 20 GB for a 2 hour concert) is then 'moved' to an archive area on the RAIDed NAS.

I suppose the big difference between our ways of working is that I have made a knowing decision that a job is a job, and when it's done, my source material is largely abandoned. With my "customers", I know they won't want a new variant x months down the line!

Incidentally, I rarely if ever have two jobs recorded on the same date. So my shorthand way of telling what files belong to what is to use a 5-numeric code YYMDD; for last Saturday's concert, for example, the source from camera 1 when imported into MEP was named 17708Cam#1, the project file like 17708.MVP, and so forth. Those of you who are still awake at this stage will wonder how I deal with October, November and December - I lapse into hex, so Dec 08 this year becomes 17C08.

I hope this was helpful, or if not, at least interesting.

Best Wishes  - Mike -


terrypin wrote on 7/11/2017, 2:33 PM

Thanks Mike, much appreciate that detailed description of your workflow and structure.

I reckon any ten of us would have ten different approaches to backup, but here are the key aspects of mine.

- My projects are mainly family-based. These days mainly holidays, usually walking. So lots of map stuff complementing the...

- Photos and movie clips, currently largely from iPhone 6S+

- I horde that stuff as I'm never sure that I'll never need it again. So it's on my 4TB HD in folders separate from my projects. And gets backed up nightly and weekly, as do all my data and Program Data, to three external USB HDs, total capacity 6 GB. Actually that's not quite accurate; periodically I delete massive chunks of old MOV files.

- Some data is in my Dropbox folder, on my 256 GB SSD, so accessible on iPad and iPhone. And providing a further level of backup and earlier file versions. But as DB capacity is 'only' 100 GB, my photos, videos and MEP projects are on the HD.

- When a project is finished I keep an MP4, a DVD folder, and sometimes these days an MEP backup of the project and its media. I then delete all the project except my Notes.txt and a folder called DVD inserts, similar to yours. An Excel spreadsheet includes details of all projects, with hyperlinks to: the project folder; the MP4 video, if there is one, for wireless streaming to lounge TV downstairs; the DVD folder; the DVD front/rear cover images; and an image of the DVD menu.

- So the project gets mostly deleted, but photos, videos, and all GPS/GPX/KML data etc used in it remains on my PC - until storage limitations dictate otherwise!

Terry, East Grinstead, UK

Terry, East Grinstead, UK. PC: i7 6700K, 4.0 GHz, 32GB with Win 10 pro. Used many earlier versions of MEPP, currently mainly MEPP 2016 & 2017 (Using scores of macro scripts to add functionality, tailored to these versions.)

Recycler wrote on 7/11/2017, 3:06 PM

 

I reckon any ten of us would have ten different approaches to backup

Definitely - any ten would more like have fifteen or more! - Mike -

browj2 wrote on 7/11/2017, 3:41 PM

Arrrgh! I just lost another long message getting booted out. The forum must not like long messages made on my BlackBerry tablet.

So, I will only say that I also copy all photos, videos and occasionally projects onto BluRay data disks.

John CB

John C.B.

VideoPro X(17U); Movie Studio 2025 Platinum; Music Maker 2025 Premium Edition; Samplitude Pro X8 Suite; see About me for more.

Desktop System - Windows 11 Pro 24H2; MB ROG STRIX B560-A Gaming WiFi; Graphics Card Zotac Gaming NVIDIA GeForce RTX-3060, PS; Power supply EVGA 750W; Intel Core i7-10700K @ 3.80GHz (UHD Graphics 630); RAM 32 GB; OS on Kingston SSD 1TB; secondary WD 2TB; others 1.5TB, 3TB, 500GB, 4TB, 5TB, 6TB, 8TB, 12TB, 14TB; three monitors - HP 25" main, LG 4K 27" second, HP 27" third; Casio WK-225 piano keyboard; M-Audio M-Track USB mixer.

Notebook - Microsoft Surface Pro 4, i5-6300U, 8 GB RAM, 256 SSD, W10 Pro 20H2.

YouTube Channel: @JCBrownVideos

terrypin wrote on 7/11/2017, 4:17 PM

Thanks John - and commiserations again. (Hope you saw my previous reply on that annoyance.)

Terry, East Grinstead, UK. PC: i7 6700K, 4.0 GHz, 32GB with Win 10 pro. Used many earlier versions of MEPP, currently mainly MEPP 2016 & 2017 (Using scores of macro scripts to add functionality, tailored to these versions.)