PowerSDR 2.8.154 and 2.7.2 lockups with TI chipset Firewire

[N3FIX]

I haven't had this Flex 5000A very long. I've recently put it in the forefront of my shack and been trying to depend on it.
I was excited to go to PowerSDR 2.8.x with the new features, however it seems to want to lock up on me. To make sure it wasn't the PowerSDR version, I went back to the 2.7.2 version, and it still locks up. The DttDSP auto restart feature in 2.8.x is a blessing, but doesn't solve the fundamental problem.

These are the things that I've done:
- Originally it seemed to lock up when Google Chrome was opened, so I got a new Firewire card... that interaction has seemed to stop.
- Got a new Firewire card with the recommended TI chipset.
- Put ferrites around the Firewire cable (which I believe is good quality).
- Using the default TI chipset firewire drivers on Win 8.1 it locks up randomly, but will re-start if the check box is ticked in 2.8.x for DttDSP auto restart.
- I've loaded the Legacy Firewire drivers, but then when it locks up it stays locked up.
- I went back to the TI drivers and I've been TRYING to live with it, however when it locks up mid-QSO it does get annoying.
- More often than not, the lockup occurs during RX not when transmitting.
- I took some snapshots of the Event Log. I'm not smart enough with .NET stuff to recognize if these are significant, or how to interpret the messages. They seemed to correspond with the timestamp of the lockup event.

So I'm looking for recommendations as to what to do to make this 5000A transceiver as reliable as my Icom. It does appear that it is not strictly a 2.8.x problem, but this is apparently where the experts hang out. I'm not sure where to look for clues as to the cause.
-
-

Windows Event Log

PowerSDR-lockup.png (44.54 KiB) Viewed 23202 times

After installing the TI Chipset...

Event log with TI chipset

2020_03_20_08_55_51_Event_Viewer.png (86.77 KiB) Viewed 23202 times

[ke9ns]

try doing the PERFMON trick first.
Click Start, type "cmd" in the Search box.
Right click cmd.exe, and select Run as administrator.
At the prompt, type lodctr /R and press ENTER. (note the "R" is a capital letter)

Just googling I found another Win8 user with the exact same Perflib Error flooding occurring (unrelated to PowerSDR)

If its happening under v2.7 and v2.8 with a TI firewire chipset (XIO2213) that has a 1394b connector, then my guess is something else is happening on your Win8 computer in the background (unrelated to PowerSDR)?

Also, How many CPU cores/threads do you have?
Also, open your task manager and look to see if anything is slamming your Hard drive, or CPU (running in the background)


Darrin

[N3FIX]

Thank you, Darrin.

As it was affecting both 2.7.2 and 2.8.x of PowerSDR, I figured it was something else.

I got a command line response after issuing the command.

Code: Select all

Info: Successfully rebuilt performance counter setting from system backup store

The PC is a Dell Optiplex 990 with i5-5200 (4 cores) @ 3.3Ghz with the max of 16GB of RAM installed. The CPU usage isn't pegged on anything and normally runs around 15% CPU. I just watched the task manager while PowerSDR locked, and the CPU trend never really changed. The hard drive isn't banging away.

I watched it for half an hour, then it began auto-restarting over and over. I had to shut down the application. No unusual CPU activity. This time there were no Events in the log that matched the timestamp.

Any other ideas where to look?

Thanks
Eric
N3FIX

[ke9ns]

Ok on the CPU.
4 cores should work perfectly with v2.7.2.
You will see a higher CPU running v2.8, but it should still work.

If the Task manager does not show any programs using the CPU or HDD, then it must be a DPC Latency issue.

The PowerSDR DttSP lockup does not always happen due to a slammed CPU or HDD, it can happen when a PC background task hogs too much time (because the DttSP routine needs to run in real-time)

You can run the "FlexRadio" app (in windows control panel) to check the DPC latency the PC is currently experiencing,

and You can google for a free DPC latency checker to try and find which background task is causing it.
I know of "LatencyMon" OR "IDLT" (in depth latency test).
"Process Explorer" will show you every process thats running in the background (you wont believe how many things are actually running in the background on a typical Win pc, yikes.)


Darrin

[N3FIX]

I started running LatencyMon. I turned off the Intel Speed Stepping feature in the BIOS. I never had these type of issues exclusively running Unix/Linux operating systems the last 20 or so years. Windowz and their jargon is rather new to me, and frankly I loathe it.

Here are the results of LatencyMon frozen at the point where PowerSDR locked up permanently:
_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be suitable for handling real-time audio and other tasks without dropouts.
LatencyMon has been analyzing your system for 0:05:15 (h:mm:ss) on all processors.
_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name: N3FIX-WIN8
OS version: Windows 8.1 , 6.3, build: 9600 (x64)
Hardware: OptiPlex 990, Dell Inc., 06D7TR
CPU: GenuineIntel Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
Logical processors: 4
Processor groups: 1
RAM: 16340 MB total
_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed: 3293 MHz

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.

WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues. It is suggested that you run a utility which reports your actual CPU frequency and temperature.
_________________________________________________________________________________________________________
MEASURED INTERRUPT TO USER PROCESS LATENCIES
_________________________________________________________________________________________________________
The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service routine started execution. This includes the scheduling and execution of a DPC routine, the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.

Highest measured interrupt to process latency (µs): 222.992138
Average measured interrupt to process latency (µs): 3.940810

Highest measured interrupt to DPC latency (µs): 162.034733
Average measured interrupt to DPC latency (µs): 2.017750


_________________________________________________________________________________________________________
REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 38.356514
Driver with highest ISR routine execution time: ataport.SYS - ATAPI Driver Extension, Microsoft Corporation

Highest reported total ISR routine time (%): 0.126595
Driver with highest ISR total time: ataport.SYS - ATAPI Driver Extension, Microsoft Corporation

Total time spent in ISRs (%) 0.212855

ISR count (execution time <250 µs): 1302104
ISR count (execution time 250-500 µs): 0
ISR count (execution time 500-999 µs): 0
ISR count (execution time 1000-1999 µs): 0
ISR count (execution time 2000-3999 µs): 0
ISR count (execution time >=4000 µs): 0
_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.

Highest DPC routine execution time (µs): 621.914364
Driver with highest DPC routine execution time: Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft Corporation

Highest reported total DPC routine time (%): 1.001634
Driver with highest DPC total execution time: Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft Corporation

Total time spent in DPCs (%) 1.222925

DPC count (execution time <250 µs): 1253704
DPC count (execution time 250-500 µs): 0
DPC count (execution time 500-999 µs): 1
DPC count (execution time 1000-1999 µs): 0
DPC count (execution time 2000-3999 µs): 0
DPC count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: chrome.exe

Total number of hard pagefaults 54
Hard pagefault count of hardest hit process: 27
Number of processes hit: 7
_________________________________________________________________________________________________________
PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s): 23.740849
CPU 0 ISR highest execution time (µs): 38.356514
CPU 0 ISR total execution time (s): 2.682909
CPU 0 ISR count: 1302104
CPU 0 DPC highest execution time (µs): 621.914364
CPU 0 DPC total execution time (s): 15.342197
CPU 0 DPC count: 1224693
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s): 0.564081
CPU 1 ISR highest execution time (µs): 0.0
CPU 1 ISR total execution time (s): 0.0
CPU 1 ISR count: 0
CPU 1 DPC highest execution time (µs): 43.068327
CPU 1 DPC total execution time (s): 0.020125
CPU 1 DPC count: 11802
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s): 0.571105
CPU 2 ISR highest execution time (µs): 0.0
CPU 2 ISR total execution time (s): 0.0
CPU 2 ISR count: 0
CPU 2 DPC highest execution time (µs): 51.461889
CPU 2 DPC total execution time (s): 0.015793
CPU 2 DPC count: 7669
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s): 0.620647
CPU 3 ISR highest execution time (µs): 0.0
CPU 3 ISR total execution time (s): 0.0
CPU 3 ISR count: 0
CPU 3 DPC highest execution time (µs): 76.796842
CPU 3 DPC total execution time (s): 0.036120
CPU 3 DPC count: 9541
_________________________________________________________________________________________________________

[N3FIX]

This morning PowerSDR.exe actually stopped on these faults:

Faulting application name: PowerSDR.exe, version: 2.8.0.155, time stamp: 0x5e72c894
Faulting module name: KERNELBASE.dll, version: 6.3.9600.19651, time stamp: 0x5e44d95d
Exception code: 0xe0434352
Fault offset: 0x00034e28
Faulting process id: 0xd708
Faulting application start time: 0x01d5ff86d6327a0b
Faulting application path: C:\Program Files (x86)\FlexRadio Systems\PowerSDR v2.8.0\PowerSDR.exe
Faulting module path: C:\Windows\SYSTEM32\KERNELBASE.dll
Report Id: 608d6d82-6b7c-11ea-825e-180373ca8ff1
Faulting package full name:
Faulting package-relative application ID:

-----------------------

Application: PowerSDR.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException
at System.Data.RBTree`1+RBTreeEnumerator[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Data.Index.InitRecords(System.Data.IFilter)
at System.Data.Index..ctor(System.Data.DataTable, System.Data.IndexField[], System.Comparison`1<System.Data.DataRow>, System.Data.DataViewRowState, System.Data.IFilter)
at System.Data.Select.CreateIndex()
at System.Data.Select.SelectRows()
at System.Data.DataTable.Select(System.String)
at PowerSDR.DB.SaveVars(System.String, System.Collections.ArrayList ByRef)
at PowerSDR.Setup.SaveOptions()
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

[ke9ns]

OK. Very strange.

Its possible something in the database may have become corrupted.
If you cannot run PowerSDR, go to the folder: %userprofile%\AppData\Roaming\FlexRadio Systems\PowerSDR v2.8.0\
You can remove the (5) database files (with your radios serial# in the file name).
Start with a fresh database and see what happens?

Darrin

[N3FIX]

I cleared the database and updated to 2.8.157

(I'm still running the Legacy firewire drivers.)

After starting with a fresh database, it ran for a couple hours and then locked up flashing the Start / Stop because I had the "DttSP Auto Restart" selected.

To bring it out of this condition I close PowerSDR, then go to the DeviceManager. I disable then immediately re-enable the Firewire driver. Then it comes up and runs for a time before doing it again. Is this a clue to the source of the problem?

[ke9ns]

If you get the flashing from the DttSP auto Restart, then that indicates the FireWire connection was lost. (that requires you to close PowerSDR)
(i.e. driver crash, bad connection in the FireWire Cable connector, OR Power to the Flex radio was last, etc.).
Assuming your Power Supply to the Radio is Solid, there is still the driver or cable.

The only time I have seen that happen, was when I reached behind the radio and accidentally hit or pull the 1394A cable.

So if you see that happen and your using a FireWire identical to the one I am using (TI firewire chipset (XIO2213)), I would first go back to the TI driver since its working on my Win10 computer, then I would try a different FireWire cable and/or try the other 1394A port on the Flex-5000 and card.

As another test, try running PowerSDR with the setup->Audio->Primary->Sample Rate = 96khz
I have seen some FireWire Cards or drivers that have troubles at 192khz SR.

as a Last resort, I would put the FireWire card into a different PC and try that, just to eliminate the different pieces of the puzzle.


Darrin ke9ns

[N3FIX]

We are making some progress. Since I'm working from home I get to do a LOT of testing by watching / operating the radio all day long.

Yesterday I changed to 96000 on the Primary sample rate, while running the Legacy Driver. That didn't make any difference. It still locked up, and when it did I had to enable/disable the Firewire Card to get it out of that state.

I un-installed Google Chrome. I think this was a big contributing factor to taking the CPU time. It would still glitch on DttSP fairly often and randomly totally lock up.

I changed it back to 192000 without making it much worse. Then I set the Process Priority to HIGH in the Setup menu of PowerSDR.
Then I re-loaded the TI driver for the firewire card. I've been running since 7 am this morning with those settings and its only glitched a couple times.

So far no total lockups. *Touch wood*
Thank you for all the suggestions, I think this may be something I can live with.
All the feature improvements in 2.8.x versions are superb.

73

N3FIX

[wd5jko]

This is a long shot, but I had DDS lockups, and the occasional overall lockup. Turned out that my F5K was an early serial number (2007 build date), and there were possible firmware timing issues that was solved by terminating a trace within the F5K with a 50 ohm resistor. The process was very difficult to do. Afterward, those issues mentioned went away completely.

I attach the Flex two page ECO here.

Jim
Wd5JKO

[ke9ns]

Jim, Thanks for providing the documents. Thats a great resource for everyone.

Darrin