BDDC Example

[jeremylt]

Nothing much to look at here yet. I'm just putting this here to make it easier to see/comment on what I'm doing.

[jeremylt]

There is some debugging to do after #744 merges, but this should be close. The open task is prolonging into the broken space - We don't really have a way to go from 1 qfunction output into a pair of target vectors

[jeremylt]

I have not started debugging yet, but all of the pieces I want are there. Here's to hoping

[jeremylt]

Huzzah - it compiles. Now to see what else is broken about it.

[jeremylt]

The number of iterations grows less rapidly when I increase the number of cells though, so that's a win

[stefanozampini]

@jeremylt @jedbrown I did not know you got a working BDDC code. I was wondering if the Arrinv solve can be hooked up in PCBDDC as subdomain solver

[jedbrown]

Are you wanting to use PCBDDC with one subdomain per process and put libCEED's Arrinv as subdomain solver? Or would you make PCBDDC work with many subdomains with hooks so that libCEED is responsible for (batched) matrix-free operators?

I would really like to use adaptive coarse basis construction in our framework, still with separable element solves.

[stefanozampini]

The idea would be to reuse PCBDDC code, and hook up your specialized solvers for interior and Arr. Adaptive coarse spaces can be built provided we have the explicit Schur complement. I have an old branch where I started supporting multiple subdomains per process https://gitlab.com/petsc/petsc/-/tree/stefanozampini/bddc-ceed. I'm also interested in this

[jeremylt]

Note: This code mostly works. There is some small bug that is killing our convergence that I haven't had space time to chase down.

[stefanozampini]

Note: This code mostly works. There is some small bug that is killing our convergence that I haven't had space time to chase down.

Getting BDDC to work can be painful, I know that :-)

[jeremylt]

Rebased for changes on main. Same slow convergence, but it does converge.

-- CEED Benchmark Problem 3 -- libCEED + PETSc + BDDC --
  PETSc:
    PETSc Vec Type                     : seq
  libCEED:
    libCEED Backend                    : /cpu/self/xsmm/blocked
    libCEED Backend MemType            : host
  Mesh:
    Number of 1D Basis Nodes (p)       : 3
    Number of 1D Quadrature Points (q) : 4
    Global Nodes                       : 125
    Owned Nodes                        : 125
    DoF per node                       : 1
  BDDC:
    Injection                          : scaled
    Global Interface Nodes             : 8
    Owned Interface Nodes              : 8
  KSP:
    KSP Type                           : cg
    KSP Convergence                    : CONVERGED_RTOL
    Total KSP Iterations               : 26
    Final rnorm                        : 1.132970e-09
  BDDC:
    PC Type                            : shell
  Performance:
    Pointwise Error (max)              : 4.185030e-02
    CG Solve Time                      : 0.0201494 (0.0201494) sec

[jedbrown]

How does the iteration count and condition number (`-ksp_view_singularvalues`) vary under grid refinement?

[jeremylt]

Size	Its	Singular Values
3x3x3	26	max 12.0304 min 0.305409 max/min 39.3913
5x5x5	64	max 12.2401 min 0.264016 max/min 46.3613
10x10x10	79	max 12.4294 min 0.108125 max/min 114.954
15x15x15	110	max 14.8259 min 0.0588987 max/min 251.719

[jedbrown]

This is for 3D with only corners as primal dofs?

[jeremylt]

Correct, 3D 2nd order basis as the fine mesh with corners only as the vertex space

[jedbrown]

Can you compare with PCBDDC using src/ksp/ksp/tutorials/ex59.c?

[stefanozampini]

Are you using exact solves? If so, the minimum eigenvalue must be equal to one.

[jedbrown]

@jeremylt I think we need to compare with ex59, or use the new COO assembly (as in Ratel) to compare with PETSc BDDC (one subdomain per process, but you can run 64 processes on Noether). Matching minimum eigenvalues is important. If the condition number table above still applies on this branch, then we're still missing something.

[jeremylt]

No idea why it changed

Size	Its	Singular Values
5x5x5	89	max 50.0355 min 0.176477 max/min 283.524
10x10x10	149	max 50.0376 min 0.161565 max/min 309.706
20x20x20	188	max 50.2384 min 0.0984236 max/min 510.43

[jeremylt]

Oh cool, some memcheck backend change shows that we have a problem with how we are treating memory. That gives some direction to future debugging efforts here

[jeremylt]

ToDo: Fix assembly to use libCEED native assembly

[jeremylt]

Turns out the memcheck issue was unrelated - see the vector fix PR.

I wonder if the convergence issue can in part can be related to the loss of accuracy from using coloring to assemble the schur problem, which is part of why I want to switch to libCEED assembly (and it'll be faster) Nope, I assembled the way I assembled for good reasons since I'm assembling the action of several operators in sequence with other actions in between.

[jeremylt]

Note to self - I'm suspicious about possible pollution of helper function input vectors. Also, the addition of the preconditioning contributions back from the restricted problem looks funko