Loading GSL code broken in recent commit

[bheatwole]

Sometime since commit b260401 the following script breaks:

.#  lookup_code_generator.gsl
.#
.#  Generates code files for lookups from the messaging model
.#
.gsl from "/project/doc/common_functions.gsl"
.#
.#

The command passed to gsl is:

gsl -script:lookup_code_generator /project/doc/model/message_model.xml

The output is:

2015/03/03 13:20:58: GSL/4.1a/github Copyright (c) 1996-2011 iMatix Corporation
2015/03/03 13:20:58: gsl/4 I: Processing /project/doc/model/message_model.xml...
2015/03/03 13:20:59: (lookup_code_generator.gsl 5) File:  not found

Note that there are two spaces after File:. Looking at the commit log, I suspect the name of the file is being freed before it can be used, but I haven't tried to debug it.

Broken with: 91b9c6d
Last known good commit: b260401

[bheatwole]

After further testing, here is the status for the recent commits:

91b9c6d - broken
89c6b06 - okay
86872fd - broken
b260401 - okay

[jschultz]

I can't replicate this problem - this code works as expected for me under Linux. Can you tell me what system you are working on?

Possibly the same bug as #73

[jschultz]

I've just given up trying to build GSL under Windows - I have MSVS 2013, which apparently is not supported. Tried using Gnu make with Makefile, but still have problems with pcre. Might be worth an option to build gsl without pcre support?

Maybe someone else can try finding the source of this problem - presumably you'll find that file_open (in sflfile.c) is being called with a null filename. If you can get a backtrace at that point we might be able to work out what has happened.

[bheatwole]

I actually compiled using a Dockerfile. This is the version that works:

FROM ubuntu:14.04

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    build-essential \
    libpcre3-dev \
    git

RUN cd /tmp && \
    git clone git://github.com/imatix/gsl && \
    cd gsl/src && \
    git checkout b26040196cc595cb5a554f14ca9a824e0c887721 && \
    make && \
    make install
RUN rm -rf /tmp/gsl

CMD ["/usr/local/bin/gsl"]

Take out the git checkout line to test the latest commit

The OS is CoreOS 604.0.0 running on Vagrant via VMWare Workstation. The underlying OS is Windows. I've been using this setup successfully for some months, but I re-ran the Dockerfile recently and that's when it broke.

[evoskuil]

The VS2013 build is only not supported because the PCRE NuGet package isn't available for VS2013 (vc120). Given that PCRE sources are included with GSL it might be better to just eventually include them in the Windows build directly. I think I tried that at one point, but don't remember the problem. Otherwise I could probably package a version of pcre_vc120 in my own NuGet repo. I just didn't see much reason for doing this since most people use GSL as a program and not a library.

VS2012 express is available as a free download from MSFT, this is how I use it (and VS2010).

[evoskuil]

Also, this break isn't recent, as I was using the NuGet packaged version of GSL that I made some time ago. I did update it locally just to confirm that the problem still exists.

[jschultz]

@bheatwole I don't have a problem getting a version that works, just one that reproduces the bug(s). I use Debian as my native OS, and those commands are pretty much equivalent to what I do anyway.

@evoskuil OK so it does seem to be a Windows problem. I'll try the free VS2012 version. But don't let me stop anyone else from producing that backtrace.

[evoskuil]

@jschultz I figured as much, otherwise there would be more complaining around here :).

[bheatwole]

I tried it on a pure linux system and all commits work there, so it probably is a Windows issue.

[jschultz]

Indeed. And the fact that it concerns file operations makes me suspect that it is related to #73 which also appears to only affect Windows.

[evoskuil]

Hmm, my last two comments pertain to #73. Forgot which issue thread I was on.

[evoskuil]

@bheatwole You can probably work around by naming the script inside of the XML. I do that on Windows and it works fine:

<?xml version = "1.0" ?>
<generate script="lookup_code_generator.gsl">
...
</generate>

and

gsl -q /project/doc/model/message_model.xml

I assume this would bypass the filesystem issue on Windows because the GSL command line is processing the files, not the GSL script processor.

[jschultz]

OK I'll resist the temptation to complain about taking longer to install VS2012 than it would take to install an entire Linux OS, and focus on the more immediate problem of why it still won't build gsl. I just get this:

Error   1   error : This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ..\..\..\..\..\..\nuget\pcre.redist.8.33.0.1\build\native\pcre.redist.targets. C:\Users\Jonathan\src\gsl\builds\msvc\vs2012\gsl\gsl.vcxproj    376 5   gsl

even when I enable nuGet. Can anyone help me here?

[evoskuil]

Open the solution, open the NuGet Package Manager (right click on project) and when the GUI finally opens you should see a band at the top of the Window with a notice to enable download of the missing package.

Otherwise you could just run builds\msvc\buildbuildall.bat, which will pull missing packages.

[evoskuil]

See also: https://github.com/imatix/gsl/blob/master/builds/msvc/README.md or more specifically: https://github.com/imatix/gsl/blob/master/builds/msvc/NUGET.md.

[jschultz]

Hmmm none of this seems to work for me. I've tried right-clicking everywhere I can but can't find anything called 'NuGet Package Manager' (maybe you can send me a screenshot?). Tried the buildall.bat script but it also fails instantly and when I look in the logfiles I see similar error message as when I tried building with the GUI. I tried following the README files and met similar dead-ends (manual download of NuGet package manager produces a file that the OS has no more idea than I do what to do with). What am I missing?

[evoskuil]

Use the NuGet Package Manager to download and update packages.

[evoskuil]

Use the VS Extension Manager to install the NuGet Package Manager :).

[jschultz]

4/03/2015 11:49:22 AM - Microsoft VSIX Installer
4/03/2015 11:49:22 AM - -------------------------------------------
4/03/2015 11:49:22 AM - Initializing Install...
4/03/2015 11:49:22 AM - Extension Details...
4/03/2015 11:49:22 AM -     Identifier      : NuPackToolsVsix.Microsoft.67e54e40-0ae3-42c5-a949-fddf5739e7a5
4/03/2015 11:49:22 AM -     Name            : NuGet Package Manager
4/03/2015 11:49:22 AM -     Author          : Microsoft Corporation
4/03/2015 11:49:22 AM -     Version         : 2.8.50926.602
4/03/2015 11:49:22 AM -     Description     : A collection of tools to automate the process of downloading, installing, upgrading, configuring, and removing packages from a VS Project.
4/03/2015 11:49:22 AM -     Locale          : en-US
4/03/2015 11:49:22 AM -     MoreInfoURL     : http://docs.nuget.org/
4/03/2015 11:49:22 AM -     InstalledByMSI  : False
4/03/2015 11:49:22 AM -     SupportedFrameworkVersionRange : [4.0,4.5]
4/03/2015 11:49:22 AM - 
4/03/2015 11:49:22 AM -     SignedBy        : Microsoft Corporation
4/03/2015 11:49:22 AM -     Certificate Info : [Subject]
  CN=Microsoft Corporation, OU=MOPR, OU=OPC, O=Microsoft Corporation, L=Redmond, S=Washington, C=US

[Issuer]
  CN=Microsoft Code Signing PCA, O=Microsoft Corporation, L=Redmond, S=Washington, C=US

[Serial Number]
  33000000CC776171D4B17BA1280001000000CC

[Not Before]
  23/04/2014 3:39:04 AM

[Not After]
  23/07/2015 3:39:04 AM

[Thumbprint]
  9DD77A79F7124D8DB128A526833CF3760FCB7794

4/03/2015 11:49:22 AM -     Supported Products : 
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.Pro
4/03/2015 11:49:22 AM -             Version : [10.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.Pro
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.IntegratedShell
4/03/2015 11:49:22 AM -             Version : [10.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.IntegratedShell
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.VWDExpress
4/03/2015 11:49:22 AM -             Version : [10.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.VWDExpress
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.VPDExpress
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.VSWinExpress
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM -         Microsoft.VisualStudio.VSWinDesktopExpress
4/03/2015 11:49:22 AM -             Version : [11.0]
4/03/2015 11:49:22 AM - 
4/03/2015 11:49:22 AM -     References      : 
4/03/2015 11:49:22 AM - 
4/03/2015 11:49:22 AM - Searching for applicable products...
4/03/2015 11:49:22 AM - Failed to initialize settings for Isolated Shell C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\ssms.exe: The system cannot find the path specified. (Exception from HRESULT: 0x80070003)
4/03/2015 11:49:22 AM - Found installed product - Microsoft Visual Studio Professional 2012
4/03/2015 11:49:23 AM - System.IO.DirectoryNotFoundException: The system cannot find the path specified. (Exception from HRESULT: 0x80070003)
  at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
  at Microsoft.VisualStudio.Settings.ExternalSettingsManager.GetScopePaths(String applicationPath, String suffixOrName, String vsVersion, Boolean isLogged, Boolean isForIsolatedApplication)
  at Microsoft.VisualStudio.Settings.ExternalSettingsManager.CreateForApplication(String applicationPath)
  at VSIXInstaller.App.GetExtensionManager(SupportedVSSKU sku)
  at VSIXInstaller.App.GetExtensionManagerForApplicableSKU(SupportedVSSKU supportedSKU, IInstallableExtension installableExtension, List`1 applicableSKUs)
  at VSIXInstaller.App.InitializeInstall()
  at System.Threading.Tasks.Task.InnerInvoke()
  at System.Threading.Tasks.Task.Execute()

[evoskuil]

Yeah... don't know about that. Did you invoke it from the VS2012 Extension Manager UI?

[evoskuil]

You should be able to manually download the PCRE NuGet packages and extract them to the proper location. This avoids the need for the NuGet Package Manager Extension.

http://www.nuget.org/packages/pcre
http://www.nuget.org/packages/pcre.redist

To do this you first need to install NuGet (not the extension).

[jschultz]

When I follow that link I just end up somewhere I've already been, ie here: http://visualstudiogallery.msdn.microsoft.com/27077b70-9dad-4c64-adcf-c7cf6bc9970c

[jschultz]

Perhaps a problem with having both VS2012 and VS 2013 installed?

[evoskuil]

No, that's not a problem. I use 2010, 2012 and 2013 concurrently.

[evoskuil]

http://docs.nuget.org/consume/installing-nuget (See bullet 5 - command line tool).

[jschultz]

OK so 'Manage NuGet packages' appeared on the pull-down menu, once. When I select it, I get an error box saying (FFS I can't even copy the text from the error box - every time I use Windows I am reminded how much I hate it) 'Operation failed The project 'gsl' is unsupported'.

Do you use Teamviewer? My Windows box is sandboxed anyway so I'm happy to let you play with it.

[evoskuil]

Sure... https://pgp.mit.edu/pks/lookup?search=eric%40voskuil.org&op=index

[evoskuil]

Also, I'm going to update the repository expectations in the GLS solutions so they use a local NuGet repo. Will make it easier, and given it's a standalone app the shared repo isn't so important.

[jschultz]

OK I've some weird things that may explain the strange behaviour around the file functions under Windows. Line 204 of sfldir.c:

dir-> file_name = dir-> _dir_entry.cFileName;

is not doing what it looks like it should do. Putting a (char *) cast on the assignment fixes that one.

Next gets even weirder. At line 483 of ggfile.c,

directory-> path      = xstrcpy (NULL, dirst-> dir_name, "/", NULL);

the arguments to xstrcpy (which are iterated with va_start), get lost inside xstrcpy, ie srcptr shows a value of 0xffffffff.

I'm afraid I'm going to have to put this down now, as I too many other calls on my time.

[evoskuil]

I've picked it up. Working through issues. Found another case of the char * assignment problem, which I don't yet have an explanation for. Also an improper truncation of the directory_open() directory (not just Windows).

[jschultz]

If there is a typing problem can we tweak the compiler options so that it gets picked up during compilation?

[evoskuil]

I'm not sure, it appears that it may be incorrect presentation in the debugger. It's not happening consistently, and can't make sense of it otherwise. The segment stripping is a real issue though.

[evoskuil]

@jschultz The string assignment problem you mention above occurs in at least one other place. It still happens with a (char *) cast or a &path[0] dereference. Sometimes it's fine without either. Definately not just debugger presentation, it causes a hard failure in later code. It's a Heisenbug.

[evoskuil]

I wasn't able to fully explain the string assignment issue, but I was able to eliminate it by disabling the UNICODE symbol definition. This affects a large number of definitions and probably somewhere the code is calling a UNICODE (W) vs. ANSI (A) API or similarly using an type defined for UNICODE. It would be better to fix that, but at least we know that's an issue.