SSDs and improved performance in MEP

Scenestealer wrote on 1/21/2015, 12:55 AM

I have started a new thread about this discussion in this thread: http://www.magix.info/us/using-last-templates.forum.1119765.html

This is copied from discussions I had with Ralf from support, a little while ago that stemmed from his comment: Your CPU is still where the power really comes from and maybe you already have or you haven't invested into other ways of boosting the general performance of your system. I'm thinking mostly of operating from SSD drives instead of conventional HDDs.

To which I replied:

I am interested in your comment about performance improvements with SSD’s. In what configuration would I expect to see a benefit? Most of the forum members have seen little improvement in MEP when using these as C: drives and that would seem to be logical given the fact that my performance monitors show little activity on this drive during preview (I have my video data on a separate HDD and my pagefile / swapfile on another HDD, however). There seems to be a lot of reading and writing to the Page file though and so I assume this is where MEP creates a cache, so would I benefit from just using a small SSD to place the page file on?

Performance boost through SSD use: If you can get your swapfile onto your SSD capacities as well, you should definitely do so. I would never really consider this a separate entity from the OS itself. If you want good OS performance, make the swapping fast as well. Naturally, the more goes onto SSD, the better. But I understand that most people would consider it a luxury at this point to also have the projects/source material on SSD. If you're willing to do a lot of moving of files etc. when you're about to start a new project or have finished a project, you could completely work from the SSD, of course.

SSD performance boost during preview - I guess my point/question here is :- would the greatest gain be from the swapfile residing on the SSD, if the C:\ drive has the O/S, program and swapfile on it (ie a standard Windows configuration) and is an SSD?
I suggest this because I think that the program is creating a cache in virtual memory where the files are preloaded from say the E:\ source data HDD prior to processing during preview, causing a lot of simultaneous reading and writing to this virtual memory cache that would be helped by the SSD’s fast read and write performance. Is this correct? An SSD video data drive would seem to be of lesser benefit because the data read rate for compressed AVCHD (even 50P) source material would seem to be well within an HDD’s capability without causing a bottleneck. Am I correct in assuming this? Following on from this train of thought I was suggesting that MEP’s preview performance might benefit on an existing system already configured with HDD’s by just adding a small SSD as a new drive and transferring the swapfile to that drive after disabling it on the C:\ HDD. What do you think?

Regarding the whole SSD thing: Having just the swapfile on an SSD drive when everything else in that setup is still on conventional hard drives sounds like an awfully strange and unlikely configuration. Would it bring some kind of performance boost? Probably. It's a very theoretical scenario - certainly not one I have tested. And as far as the topic of source material on conventional hard drives goes - this will definitely add another performance boost to your experience. Keep in mind, it's not just the actual reading speeds and constant data throughput that can be reached in a good or average reading situation that matters. There are also relevant factors here like the much faster seeking and reaction time (e.g. relevant when transitioning from one video file to another one) that positively contribute in an SSD setup. But, as I said, the source material is really the last thing that you would put on SSD - luxury. :-)

Me again:-

After that discussion I still feel there may be a benefit purely due to improved paging speed with the Pagefile on the SSD although Ralf may have overstated the benefit of SSD's in general for MEP. Unfortunately I have not yet been able to test the theories having not upgraded my system.

My opimion is that if your system can not read and write to an HDD when rendering at 1/2 or less real time during export then it is unlikely to be previewing in realtime given the shennanigans that it requires of your system to achieve smooth playback.

Cheers

Peter

Comments

browj2 wrote on 1/21/2015, 10:42 AM

Peter,

Good initiative and thanks for the info from Magix.

1. Best location for Magix folders

I am curious about where to place the project files, HD or SSD. By default, the Magix programs create folders under My Documents which is usually the C: drive unless the user has changed the default drive for this.

Here is a link about how to change the default user folder. I didn't do this after installing my new SSD as drive C: and adding in a new HD, even though I want everything to go to the new HD. Right now I have a bit of a mess. One of the problems is changing the folders in all Magix programs. Another is that any new install would revert back to the C: drive unless the user folder is changed.

What I am wondering is, given that Magix programs appear to place many temporary files like H0, HDP, HS, 10 minute backup files, project file, amongst others, under the C:\Users\UserName\Documents\MAGIX folders, would there be any benefit of having these folders on an SSD?

2. AppData

This folder is also on the C: drive and accumulates a lot of files. Is it best to keep this on the system SSD drive or could it be moved to another HD drive, and if so how? I am presuming that Magix programs place and access a lot of data at this location, but some of it seems to be files that would be better placed elsewhere.

An example is with the 3D Maker. Any animation files are automatically placed under AppData. I believe that this includes the file that is created when a 3D title is accessed from MEP/VPX6 and modified for use on the timeline. I have found that problems occur when I export the 3D animation to another location and then exit back into MEP/VPX6. It can't find the file that it created for the modified title. I'm still not sure how it all works. However, the avi animations are quite large and just another one of the things that takes up space on a drive with limited space. The default location is C:\Users\John\AppData\Local\Xara\MAGIX 3D Maker embeded\AVI_CFX. I can change the export and save locations, but the program exchange between 3D Maker and MEP/VPX6 as a title creates something somewhere, but what and how it works, I don't know.

3. Additional SSD connected by SATA to USB

John Baker indicated that "the bottleneck on a fast machine is the SATA interface / HDD which on a single hard drive is having to constantly switch from Read to Write during rendering." I have a second SSD that I intend to install using a SATA to USB interface. Both of my internal SATA connections are now taken by the other 2 new drives. Given what Ralf told you, I would presume that putting the source information on this drive would give some performance increase. Have you or anyone tested this type of connection (SATA to USB) to see if there are any problems or increase in speed?

I will get around to installing the disk on the weekend to see what happens. Is there any way to monitor R/W speed to be able to make a comparison between R/W to a HDD and a SSD?

Recycler wrote on 1/22/2015, 4:14 AM

What an interesting discussion!

If I take John C.B.'s final question ("Is there any way to monitor R/W speed to be able to make a comparison between R/W to a HDD and a SSD?") in a general sense, the tool I use is the free ParkDale Disc Speed Tester. The web address is a bit obscure - best to find it by searching for "ParkDale Disc Speed Tester". There are also a few YouTube tutorials, although in its simplest mode it barely needs instructions. In the specific sense of comparing HDD/SSD performance when doing Magix work, I have no answer I'm afraid.

As well as optimising storage places for streams, pagefile, instructions etc, I'd also be interested in understanding how to adjust Magix (MEP 2015+ in my case) to make more use of the 16GB of RAM on my PC. Usage never gets much above 4.6 or so - once getting up to 5.2 I think. But it seems I might as well not have bothered to buy so much RAM in the first place! Would this be down simply to the nature of the work I do (4x 1920x1080 cameras, separate audio stream, lots of cutting and captions but almost no fancy fades, or heavy video corrections)?

Any thoughts? - Mike -

Scenestealer wrote on 1/22/2015, 4:16 AM

Something has corrupted my Topic title and I can not seem to get rid of it???

Peter

johnebaker wrote on 1/22/2015, 12:53 PM

Hi Peter

Fixed.

John

johnebaker wrote on 1/22/2015, 1:10 PM

Hi John CB

. . . . (SATA to USB) . . . .

This is introducing an even worse bottleneck - the actual USB speeds* are way below what a SATA bus is capable.

If you have a spare PCI slot on your m/b then use a PCI SATA card such as this (example only not a recommendation for this particular make) and mount the SSD internally.

* In theory USB3 is nearly as fast as SATA3 however the USB max data rate is shared between all devices connected to the same controller, and the sustained data rate is very dependent on the USB controller chipsets on your m/b and very rarely approaches the maximum specified,

SATA is designed for sustained high speed data transfer.

HTH

John

browj2 wrote on 1/25/2015, 1:55 PM

Hi John EB,

Thanks for the information, but admittedly, I am a bit lost.

I took a look at both USB3 to SATA and PCI to SATA for 3 Vantec products:

1. Vantec NexStar Sata to USB 3.0 adapter (which was what I bought) says up to 5 Gbps w/USB3.0

2. Vantec 2-Port SATA II 300 PCIe Host Card with RAID (UGT-ST420R) says "once installed, this host card will add two SATA ports with two independent channels. This allows support of two SATA devices at speeds of up to 3Gb/s whether it is an -- SATA hard drive or SATA device."

Vantec 2 Channel 4-Port SATA 6Gb/s PCIe Host Card (UGT-ST622) (which is faster)

I understand about the sharing problem, but I presently have 2 PCI - USB3.0 adapter cards installed, one with 2 ports and the other with 4 ports.

If I use the one with 2 ports solely for the SATA drive using the NexStar adapter, would I not get equal to better than the first PCI adapter and at least close to the second one?

Or does this part cause a major slowdown "the sustained data rate is very dependent on the USB controller chipsets on your m/b and very rarely approaches the maximum specified?" In this case I understand that it would be the USB controller chipsets on the m/b causing a slowdown.

I haven't tried the SSD using the SATA to USB3 yet.

I did run the Parkdale Test as suggested by Mike, on my existing drives. Very interesting results.

Here is the Dropbox link.

My drive J, where I keep my source video files, is about the slowest drive that I have, thus I presume that I should use one of the faster drives.

Here is the link to the Parkdale benchmark R/W tests for many different HDD's and SDD's.

johnebaker wrote on 1/25/2015, 3:53 PM

Hi John CB

Interesting figures - however there is a small gotcha when quoting read / write speeds which makes the Parkdale figures look very disappointing.

The SATA and USB specs are quoted in Mb or Gb per second, the testing software is giving results in MB or GB per sec.

A subtle difference which is actually very significant Mb = Megabits whereas MB = Megabytes the difference is about a factor of x8 to x10 depending on data block size etc.

. . . . My drive J, where I keep my source video files, is about the slowest drive that I have, thus I presume that I should use one of the faster drives . . . .

Most definitely - according to those fgures the drive J is on a USB port - I would put the data on your B drive.

Your M drive performance is poor despite being a SATA 2 (3Gb) - unless it is on a USB adapter - is this so?

The SSD drive read performance is down - I would have expected the read speed to be about 10% faster than the write.

Which drive do you have the Windows page file on?

Re using USB drives for data / exporting to - I would not do it for several reasons.

Their actual read / write speeds are not consistent.
USB drives have a habit of disconnecting / reconnecting - you would not normally see this however where you have a sustained operation over a period of time, eg rendering / exporting, it can cause errors
USB ports have a habit of hibernating / powering down without warning - I always turn mine on permanently in the power management options.

Just to complete the picture - external NAS boxes can achieve speeds of 100 MB/sec, ie close to internal HD speeds.

However this depends on their configuration and assumes you have a network which runs at 1Gb (cable not WiFi) and that the computer. NAS box and router / switch are connected with 1Gb certified patch leads - standard cat 5e cables are not certified to 1Gb and can cause issues at these speeds.

John

browj2 wrote on 1/26/2015, 7:08 PM

Hi John EB,

Thanks for the comments.

Drives J and M are both older, external and connected by USB2, not 3. M is not SATA connected. They are both pre-USB3 drives.

Drive J is a Seagate and M is Comstar. M contains older DV files that are mostly also on J. Either way, they are obviously the slowest to access.

I installed the new 256 GB SSD (drive O:) using the SATA to USB3 interface, connected to the new 4-port USB interface with nothing else attached.

Here is the Parkdale test results (drive O:).

The results show that it is slower that the other SSD that is connected directly to the SATA port (replacing my old C drive).

The new C drive is the Crucial 512GB SSD. It contains the OS, all of the program files and Windows page file.

In the Parkdale test, the sequential R/W values are all in MBytes/sec. The Random R/W values are in MBytes/sec or kBytes/sec.

The big difference between the SSD and the new SATA HDD (drive B) is in random, where sequential write speed is 37.8 MByte/sec for the SSD vs 1.2743 MBytes/s (1274.3 kBytes/sec) which is a factor of almost 30. The IOPS (input output per sec) illustrates this same difference.

Looking at the 2 SSD drives, the SATA connected SSD (C) is somewhat faster than the SATA to USB3 connected SSD (O:). The random read speed is slightly faster for the USB connected drive (maybe because it is empty). I noticed that the SATA to USB3 interface spec goes to 5Gb/s whereas the drives can go to 6Gb/s. Thus I would expect a drop in speed of about 1/6th due to the interface, which is almost the case. However, that does not explain the next part.

As to your comment about the specs being quoted in Gb or Mb and the tests in GB or MB, and taking a factor of 10 to be generous, a 6Gb/s spec for my C drive SSD (C:) should give me a R/W speed of 600MB/s, should it not? Thus drive C seems to be falling quite short of the spec. Any ideas as to why? Motherboard?

NAS - I have a WD 3TB drive on the home network. I use it for backups and exchanging files on the home network. Both the NAS and my computer are connected using the ethernet cables (6ft each).

Here are the results:

Dismal. This is the slowest HD in the system, but it does its job. Not to be used for video.

Conclusion so far is that one should use SSD's for operations that require rapid R/W operations.

Now in the case of MEP/VPX6, which parts are not dependent on R/W speed and can be put on HDD's?

Scenestealer wrote on 1/27/2015, 4:16 AM

Hi John CB

I am not sure what type of files you are generally dealing with but if you open task manager during playback of your timeline and select "Resource Monitor>Disk" during playback (or encoding) you will see the disk through put for discs that are actively reading and writing. In my case, working with 1920x1080 50P AVCHD files (average bitrate 28Mbps), I do not see much more than 5-10MBytes per second total activity, with most of it made up of read activity of each file in turn on the E:\ disk where my source files are, and a relatively large amount of write activity to C:\Page file.sys. This Pagefile activity is what made me suspect that MEP was using the virtual memory of the pagefile to create a cache as the files are read, but I could not confirm this with Ralf at support in my original discussions.

Whatever, I believe these monitors show that unless you are working with large uncompressed AVI files (where I have seen in excess of 100MBytes per second) the activity reported is insignificant with compressed video like MPEG4 / 2, AVCHD etc., and should not trouble a good (even not so good) Sata2 HDD in realtime preview, let alone encoding, where demand should be less (as I stated earlier) providing you have a good disc configuration.

As the MEP program is loaded into RAM on startup there is very little other activity on the system disc, so I believe any benefit would come from the faster read write to the pagefile (and MEP cache if it exists there), bought about by the standard windows configuration ie. OS, program, project files (excluding video files) and pagefile.sys, on C:\ if C is an SSD.

My only comments about your disc tests would be : -

The sequential read /write speed of your SSD seems quite low compared to claimed and tested results for that drive of something like 400MBytes per sec or more, but I could not say what is holding it back. It could be Motherboard related as I presume it is an older board judjing by the lack of Sata Ports?

The random in outs per sec of the SSD compared to the HDD, which has almost the same sequential as the SSD, is impressive albeit not surprising.

I have a WD MyCloud 3TB NAS for backups and it is frustratingly slow also.

Peter

johnebaker wrote on 1/27/2015, 1:28 PM

I agree with Peter - about your motherboard may be a limiting factor.

I have checked my SanDisk SSD drive using the Parkdale s/w and it returns Seq Write speed of 213.9 MBytes/sec and a Seq Read speed of 437.6 MBytes/sec - it is a SATA 3 (6GB) drive on a SATA 3 connection. Interestingly my random QD32 figures are lower than yours!

There is something definitely odd about the c: drive SSD seq read / write figures.

John

browj2 wrote on 1/28/2015, 5:16 PM

Gentlemen,

I took a look at the mb specs, dated 2010. I have 4 SATA ports with SSD on 1, HD on 2, BD/DVD on 3, and next time I open the box I'll check for other one. I seem to recall that all 4 were taken.

I monitored disk usage during an export. Huge usage of the SSD (C:), high on HD (B:) which was the destination location, and only a small amount on K: that contains some of the source files. I'll watch again when I have one that has a lot of material on one of the USB drives. I noticed that my 16GB of RAM was being used at 80-90%.

I have no idea about why the random figures are not proportional wrt the spped of your SandDisk SSD.

Scenestealer wrote on 1/29/2015, 4:28 AM

Hi John CB

Which files are showing the most activity on C:? Is C:Page file.sys one of them and the highest?

Good to see the 64bit version is making good use of most of your RAM.

Peter

browj2 wrote on 1/30/2015, 11:53 AM

Hi Peter,

I'll watch for this on the next export and get back to you.