gocql Directly scanning into a struct pointer Go

Please answer these questions before submitting your issue. Thanks!

What version of Cassandra&Gocql are you using?

[cqlsh 5.0.1 | Cassandra 3.11.9 | CQL spec 3.4.4 | Native protocol v4]

What version of Go are you using?

go version go1.15.3 darwin/amd64

What did you do?

func FindAllByName(ctx context.Context, name string) ([]*User, error) {
    itr := s.cassandra.Query(`SELECT id FROM users WHERE name = ?`, name).WithContext(ctx).Iter()
    defer itr.Close()

    users := make([]*User, itr.NumRows())
    for i := range users {
        itr.Scan(users[i].ID) // panic: runtime error: invalid memory address or nil pointer dereference
    }

    return users, nil
}

What did you expect to see?

I wanted to be able to scan data directly into struct pointer rather than creating a new variable and dereference in a loop. The one below solves the problem but it doesn't seem optimal to me. Just wondering if I am doing something wrong or is there a solution/a better way of doing this?

func FindAllByName(ctx context.Context, name string) ([]*User, error) {
    itr := s.cassandra.Query(`SELECT id FROM users WHERE name = ?`, name).WithContext(ctx).Iter()
    defer itr.Close()

    users := make([]*User, itr.NumRows())
    for i := range users {
        var user User
        itr.Scan(&user.ID)
        users[i] = &user
    }

    return users, nil
}
Asked Oct 07 '21 04:10
avatar bentcoder
bentcoder

5 Answer:

Use the Scanner interface instead via tier.Scanner()

On Mon, 18 Jan 2021 at 12:31 pm, BentCoder notifications@github.com wrote:

Please answer these questions before submitting your issue. Thanks! What version of Cassandra&Gocql are you using?

[cqlsh 5.0.1 | Cassandra 3.11.9 | CQL spec 3.4.4 | Native protocol v4] What version of Go are you using?

go version go1.15.3 darwin/amd64 What did you do?

func FindAllByName(ctx context.Context, name string) ([]*User, error) { itr := s.cassandra.Query(SELECT id FROM users WHERE name = ?, name).WithContext(ctx).Iter() defer itr.Close()

users := make([]*User, itr.NumRows()) for i := range users { itr.Scan(users[i].ID) // panic: runtime error: invalid memory address or nil pointer dereference }

return users, nil }

What did you expect to see?

I wanted to be able to scan data directly into struct pointer rather than creating a new variable and dereference in a loop. The one below solves the problem but it doesn't seem optimal to me. Just wondering if I am doing something wrong or is there a solution/a better way of doing this?

func FindAllByName(ctx context.Context, name string) ([]*User, error) { itr := s.cassandra.Query(SELECT id FROM users WHERE name = ?, name).WithContext(ctx).Iter() defer itr.Close()

users := make([]*User, itr.NumRows()) for i := range users { var user User itr.Scan(&user.ID) users[i] = &user }

return users, nil }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gocql/gocql/issues/1523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAARJPS6KTSRT2DJOQBC2DS2QS3DANCNFSM4WHHPXRA .

1
Answered Jan 18 '21 at 13:02
avatar  of Zariel
Zariel

@BentCoder your code panics before it calls Scan:

users variable is a slice of nil pointers. When users[i].ID executes, users[i] (which is nil) is derefenced, the panic occurs. So this panic is not related to gocql. The code would work if users had type []User instead of []*User (that would allocate one huge array instead of pointers, which might or might not be okay depending on how you use the results).

Note that itr.NumRows() returns the number of rows in the current page (docs), so if there are more results than page size, your code will panic on accessing out of bounds index. You could either append more items instead of assigning an index or limit the number of results or disable paging with .PageState(nil).

For example:

func FindAllByName(ctx context.Context, name string) ([]*User, error) {
    itr := s.cassandra.Query(`SELECT id FROM users WHERE name = ?`, name).WithContext(ctx).Iter()
    defer itr.Close()

    users := make([]*User, 0, itr.NumRows())
    scanner := itr.Scanner()
    for scanner.Next() {
        var user User
        err := scanner.Scan(&user.ID)
        if err != nil {
            return nil, err
        }
        users = append(users, &user)
    }
    if err := scanner.Err(); err != nil {
        return nil, err
    }

    return users, nil
}

If adding another dependency is okay with you, you could also use https://github.com/scylladb/gocqlx (see Queryx.Select).

1
Answered Jan 18 '21 at 17:20
avatar  of martin-sucha
martin-sucha

Hi @martin-sucha I get what you mean and makes sense. Just one thing that I didn't know was the itr.NumRows() comment you added which definitely had to be learned. Thanks for that.

I changed your example slightly by replacing var user User and removing append function. Also changed the make line where I had to remove the "capacity" to make this example work otherwise it will panic with index out of range [0] with length 0. Although this works for now, is this also correct implementation that avoids your code will panic on accessing out of bounds index?

func FindAllByName(ctx context.Context, name string) ([]*User, error) {
    itr := s.cassandra.Query(`SELECT id FROM users WHERE name = ?`, name).WithContext(ctx).Iter()
    defer itr.Close()

    // users := make([]*User, 0, itr.NumRows())
    users := make([]*User, itr.NumRows())

    scanner := itr.Scanner()
    for scanner.Next() {
        // var user User
        users[i] = &User{}

        err := scanner.Scan(&user.ID)
        if err != nil {
            return nil, err
        }

        // users = append(users, &user)
    }
    if err := scanner.Err(); err != nil {
        return nil, err
    }

    return users, nil
}
1
Answered Jan 18 '21 at 17:58
avatar  of bentcoder
bentcoder

Although this works for now, is this also correct implementation that avoids your code will panic on accessing out of bounds index?

This last code above won't compile. i is not defined anywhere.

I suggested the append for avoiding the out of bound access - it will reallocate the array/slice so that there is enough space for more results. The default PageSize is 5000. I recommend you try to run your code with at least 5001 user records in the result of the query.

Assuming that you have i starting from 0 and increment it at the end of each iteration, then with 5001 users, itr.NumRows() returns 5000, so in your code len(users) == 5000. When reading the last row, i == 5000 and that would panic unless you reallocate the slice to have length at least 5001.

1
Answered Jan 18 '21 at 19:18
avatar  of martin-sucha
martin-sucha

My bad, I forgot to add i there (typo). I will stick to your solution. Thank you for investing time in this. Much appreciated.

EDIT: Worth noting, I will LIMIT my query with a pagination so it should avoiding fetching thousands of record then hammer RAM!

1
Answered Jan 18 '21 at 19:29
avatar  of bentcoder
bentcoder