Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
7.198325825s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:29:38 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.198377443s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.] duration:
3.561367913s trace:
https://jaeger.dev.storj.io/trace/567cf271cf4a7926
FAIL (2025-12-08 15:29:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.561437235s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.........X.] duration:
8.189269899s trace:
https://jaeger.dev.storj.io/trace/5b539c2e670fad34
FAIL (2025-12-08 15:29:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
8:34.711858997 (71h59m34.582942032s) [upload=63035]: Starting upload
Dec 8 15:28:34.711925895 (71h59m34.58300895s) [upload=63035]: Done starting upload
Dec 8 15:28:34.712000326 (71h59m34.583083371s) [upload=63035]: Uploading segments...
Dec 8 15:28:34.712139992 (71h59m34.583223037s) [upload=63035]: Got next segment. Inline: false
Dec 8 15:28:34.712203193 (71h59m34.583286238s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:28:34.712211859 (71h59m34.583294894s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:28:35.217128953 (71h59m35.088211998s) [seg_pos={0 0}, upload=63035]: Begin upload segment...
Dec 8 15:28:35.217204688 (71h59m35.088287733s) [seg_pos={0 0}, upload=63035]: Done begin upload segment.
Dec 8 15:28:35.217223062 (71h59m35.088306097s) [noise=true , address=145.79.227.40:4051 , piece_id=5G24WVCPMQAAUSQWUE3NGAQBYCCQEJJS3MPOTJJIOT72U7PCJWTQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.217256205 (71h59m35.08833925s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.217744174 (71h59m35.088827219s) [noise=true , address=145.79.227.20:4437 , piece_id=XN2S2WEHLTWKRZO4RU55K2KYTURY3TMXVVPQHDSPFFM2CDF263RA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.218435912 (71h59m35.089518957s) [noise=true , address=77.90.31.11:4555 , piece_id=WLJ5VHR47MQZKWQBRHLOATZY33P2IFDI5P2ZZBZ25IMBCUGS4SPQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.218886059 (71h59m35.089969094s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219455743 (71h59m35.090538788s) [noise=true , address=145.79.227.21:4474 , piece_id=44X46H6W53GBKC5I7IFYEVVJK4H5DUM3CYUE4RXILPAEKOYVULXA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219692044 (71h59m35.090775089s) [noise=true , address=145.79.227.5:4221 , piece_id=CLVZGYZLSXB3J2JULQLJZLJZ5LYUYTIPAFFADSRPTLJR6GLYBCHA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219999289 (71h59m35.091082324s) [noise=true , address=145.79.227.44:4195 , piece_id=FZFCJGCEC7JGXA3QRHHWAYYWTB6WJ7ANQ3KP3F2BNXOPWWQIWG3Q, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220243624 (71h59m35.091326669s) [noise=true , address=145.79.227.27:4689 , piece_id=ZK3LAYPJNN47DKZ7FXFU4MOI45PGCL2RUP7Z34NKSIBN3KEAOGBA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220566458 (71h59m35.091649494s) [noise=true , address=145.79.227.27:4690 , piece_id=TEPT2OC5P7XBKKLQWVDLL6ISOAR4O7GKTZR25TKC5XO6WFF67SBQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220747182 (71h59m35.091830227s) [noise=true , address=145.79.227.23:4545 , piece_id=YVHM5CAIVDBCYNZMSOD4KQHO6NPVMOX47SY7ALX5FBT2FMYFOK4A, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.221295266 (71h59m35.092378311s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.221678777 (71h59m35.092761822s) [noise=true , address=145.79.227.2:4113 , piece_id=FIAB4XSDITJCYE4JQQG7NZVFPK4PE4HNIZPLUHMP4NVR6EDQJKVA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.222251147 (71h59m35.093334182s) [noise=true , address=145.79.227.43:4157 , piece_id=2LHMA33HEWNOTHQRQZVDM7HWOAW6ODFGSGL6W2QN3HWFXLEP6FPQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.222919209 (71h59m35.094002254s) [noise=true , address=145.79.227.43:4158 , piece_id=TTU4ZBQ3XQJK26WS4NPCZ22DCKVBY237RGLWTGY2HWTHUCV7ZFIQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.223485777 (71h59m35.094568813s) [noise=true , address=145.79.227.23:4547 , piece_id=EQATLNGXVGE4YFQWYDHD6GQ76D2URMBT7FNLN2CQ6G3PDIMK55UQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:37.017284127 (71h59m36.888367172s) [upload=63035]: Next returned nil segment
Dec 8 15:28:37.017299296 (71h59m36.888382331s) [upload=63035]: Waiting for error group managing segments...
Dec 8 15:28:37.361370697 (71h59m37.232453732s) [noise=true , address=145.79.227.27:4689 , piece_id=ZK3LAYPJNN47DKZ7FXFU4MOI45PGCL2RUP7Z34NKSIBN3KEAOGBA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.364707590 (71h59m37.235790625s) [noise=true , address=145.79.227.44:4195 , piece_id=FZFCJGCEC7JGXA3QRHHWAYYWTB6WJ7ANQ3KP3F2BNXOPWWQIWG3Q, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.375204949 (71h59m37.246287994s) [noise=true , address=145.79.227.20:4437 , piece_id=XN2S2WEHLTWKRZO4RU55K2KYTURY3TMXVVPQHDSPFFM2CDF263RA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.376982714 (71h59m37.248065749s) [noise=true , address=145.79.227.43:4158 , piece_id=TTU4ZBQ3XQJK26WS4NPCZ22DCKVBY237RGLWTGY2HWTHUCV7ZFIQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.400202670 (71h59m37.271285715s) [noise=true , address=145.79.227.43:4157 , piece_id=2LHMA33HEWNOTHQRQZVDM7HWOAW6ODFGSGL6W2QN3HWFXLEP6FPQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.404269715 (71h59m37.27535275s) [noise=true , address=145.79.227.23:4547 , piece_id=EQATLNGXVGE4YFQWYDHD6GQ76D2URMBT7FNLN2CQ6G3PDIMK55UQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.418700574 (71h59m37.289783619s) [noise=true , address=145.79.227.21:4474 , piece_id=44X46H6W53GBKC5I7IFYEVVJK4H5DUM3CYUE4RXILPAEKOYVULXA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.422958351 (71h59m37.294041396s) [noise=true , address=77.90.31.11:4555 , piece_id=WLJ5VHR47MQZKWQBRHLOATZY33P2IFDI5P2ZZBZ25IMBCUGS4SPQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.436109594 (71h59m37.307192639s) [noise=true , address=145.79.227.5:4221 , piece_id=CLVZGYZLSXB3J2JULQLJZLJZ5LYUYTIPAFFADSRPTLJR6GLYBCHA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.448201499 (71h59m37.319284534s) [noise=true , address=145.79.227.27:4690 , piece_id=TEPT2OC5P7XBKKLQWVDLL6ISOAR4O7GKTZR25TKC5XO6WFF67SBQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.468412445 (71h59m37.33949548s) [noise=true , address=145.79.227.23:4545 , piece_id=YVHM5CAIVDBCYNZMSOD4KQHO6NPVMOX47SY7ALX5FBT2FMYFOK4A, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.819874228 (71h59m37.690957273s) [noise=true , address=145.79.227.2:4113 , piece_id=FIAB4XSDITJCYE4JQQG7NZVFPK4PE4HNIZPLUHMP4NVR6EDQJKVA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:38.088706208 (71h59m37.959789243s) [noise=true , address=145.79.227.40:4051 , piece_id=5G24WVCPMQAAUSQWUE3NGAQBYCCQEJJS3MPOTJJIOT72U7PCJWTQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:38.088724744 (71h59m37.959807779s) [seg_pos={0 0}, upload=63035]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:28:38.088896090 (71h59m37.959979135s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 8 15:28:38.088902031 (71h59m37.959985066s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089127871 (71h59m37.960210906s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: read tcp 188.166.225.170:37544->145.79.227.16:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:28:38.089134815 (71h59m37.96021785s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089154402 (71h59m37.960237457s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection
Dec 8 15:28:38.089179560 (71h59m37.960262605s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089210268 (71h59m37.960293304s) [seg_pos={0 0}, upload=63035]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:28:38.089327862 (71h59m37.960410907s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:28:38.089335797 (71h59m37.960418832s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:28:38.409700686 (71h59m38.280783721s) [upload=63035]: Done uploading segments...
Dec 8 15:28:38.409724240 (71h59m38.280807285s) [upload=63035]: Upload finished. err: <nil>
Duration: 8.189297932s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X...X.........................................X............] duration:
28.636564804s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:29:29 UTC)
orbiter warning: context deadline exceeded
Duration: 28.636594581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [................................X............X.X.XX.XX.XX..X] duration:
30.001691486s trace:
https://jaeger.dev.storj.io/trace/397564ec35512283
FAIL (2025-12-08 15:29:22 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001718507s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
34.075713642s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:29:14 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 34.075741625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.........X] duration:
30.003849067s trace:
https://jaeger.dev.storj.io/trace/5b539c2e670fad34
FAIL (2025-12-08 15:29:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
8:34.711858997 (71h59m34.582942032s) [upload=63035]: Starting upload
Dec 8 15:28:34.711925895 (71h59m34.58300895s) [upload=63035]: Done starting upload
Dec 8 15:28:34.712000326 (71h59m34.583083371s) [upload=63035]: Uploading segments...
Dec 8 15:28:34.712139992 (71h59m34.583223037s) [upload=63035]: Got next segment. Inline: false
Dec 8 15:28:34.712203193 (71h59m34.583286238s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:28:34.712211859 (71h59m34.583294894s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:28:35.217128953 (71h59m35.088211998s) [seg_pos={0 0}, upload=63035]: Begin upload segment...
Dec 8 15:28:35.217204688 (71h59m35.088287733s) [seg_pos={0 0}, upload=63035]: Done begin upload segment.
Dec 8 15:28:35.217223062 (71h59m35.088306097s) [noise=true , address=145.79.227.40:4051 , piece_id=5G24WVCPMQAAUSQWUE3NGAQBYCCQEJJS3MPOTJJIOT72U7PCJWTQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.217256205 (71h59m35.08833925s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.217744174 (71h59m35.088827219s) [noise=true , address=145.79.227.20:4437 , piece_id=XN2S2WEHLTWKRZO4RU55K2KYTURY3TMXVVPQHDSPFFM2CDF263RA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.218435912 (71h59m35.089518957s) [noise=true , address=77.90.31.11:4555 , piece_id=WLJ5VHR47MQZKWQBRHLOATZY33P2IFDI5P2ZZBZ25IMBCUGS4SPQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.218886059 (71h59m35.089969094s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219455743 (71h59m35.090538788s) [noise=true , address=145.79.227.21:4474 , piece_id=44X46H6W53GBKC5I7IFYEVVJK4H5DUM3CYUE4RXILPAEKOYVULXA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219692044 (71h59m35.090775089s) [noise=true , address=145.79.227.5:4221 , piece_id=CLVZGYZLSXB3J2JULQLJZLJZ5LYUYTIPAFFADSRPTLJR6GLYBCHA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.219999289 (71h59m35.091082324s) [noise=true , address=145.79.227.44:4195 , piece_id=FZFCJGCEC7JGXA3QRHHWAYYWTB6WJ7ANQ3KP3F2BNXOPWWQIWG3Q, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220243624 (71h59m35.091326669s) [noise=true , address=145.79.227.27:4689 , piece_id=ZK3LAYPJNN47DKZ7FXFU4MOI45PGCL2RUP7Z34NKSIBN3KEAOGBA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220566458 (71h59m35.091649494s) [noise=true , address=145.79.227.27:4690 , piece_id=TEPT2OC5P7XBKKLQWVDLL6ISOAR4O7GKTZR25TKC5XO6WFF67SBQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.220747182 (71h59m35.091830227s) [noise=true , address=145.79.227.23:4545 , piece_id=YVHM5CAIVDBCYNZMSOD4KQHO6NPVMOX47SY7ALX5FBT2FMYFOK4A, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.221295266 (71h59m35.092378311s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.221678777 (71h59m35.092761822s) [noise=true , address=145.79.227.2:4113 , piece_id=FIAB4XSDITJCYE4JQQG7NZVFPK4PE4HNIZPLUHMP4NVR6EDQJKVA, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.222251147 (71h59m35.093334182s) [noise=true , address=145.79.227.43:4157 , piece_id=2LHMA33HEWNOTHQRQZVDM7HWOAW6ODFGSGL6W2QN3HWFXLEP6FPQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.222919209 (71h59m35.094002254s) [noise=true , address=145.79.227.43:4158 , piece_id=TTU4ZBQ3XQJK26WS4NPCZ22DCKVBY237RGLWTGY2HWTHUCV7ZFIQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:35.223485777 (71h59m35.094568813s) [noise=true , address=145.79.227.23:4547 , piece_id=EQATLNGXVGE4YFQWYDHD6GQ76D2URMBT7FNLN2CQ6G3PDIMK55UQ, seg_pos={0 0}, upload=63035]: Uploading piece...
Dec 8 15:28:37.017284127 (71h59m36.888367172s) [upload=63035]: Next returned nil segment
Dec 8 15:28:37.017299296 (71h59m36.888382331s) [upload=63035]: Waiting for error group managing segments...
Dec 8 15:28:37.361370697 (71h59m37.232453732s) [noise=true , address=145.79.227.27:4689 , piece_id=ZK3LAYPJNN47DKZ7FXFU4MOI45PGCL2RUP7Z34NKSIBN3KEAOGBA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.364707590 (71h59m37.235790625s) [noise=true , address=145.79.227.44:4195 , piece_id=FZFCJGCEC7JGXA3QRHHWAYYWTB6WJ7ANQ3KP3F2BNXOPWWQIWG3Q, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.375204949 (71h59m37.246287994s) [noise=true , address=145.79.227.20:4437 , piece_id=XN2S2WEHLTWKRZO4RU55K2KYTURY3TMXVVPQHDSPFFM2CDF263RA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.376982714 (71h59m37.248065749s) [noise=true , address=145.79.227.43:4158 , piece_id=TTU4ZBQ3XQJK26WS4NPCZ22DCKVBY237RGLWTGY2HWTHUCV7ZFIQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.400202670 (71h59m37.271285715s) [noise=true , address=145.79.227.43:4157 , piece_id=2LHMA33HEWNOTHQRQZVDM7HWOAW6ODFGSGL6W2QN3HWFXLEP6FPQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.404269715 (71h59m37.27535275s) [noise=true , address=145.79.227.23:4547 , piece_id=EQATLNGXVGE4YFQWYDHD6GQ76D2URMBT7FNLN2CQ6G3PDIMK55UQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.418700574 (71h59m37.289783619s) [noise=true , address=145.79.227.21:4474 , piece_id=44X46H6W53GBKC5I7IFYEVVJK4H5DUM3CYUE4RXILPAEKOYVULXA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.422958351 (71h59m37.294041396s) [noise=true , address=77.90.31.11:4555 , piece_id=WLJ5VHR47MQZKWQBRHLOATZY33P2IFDI5P2ZZBZ25IMBCUGS4SPQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.436109594 (71h59m37.307192639s) [noise=true , address=145.79.227.5:4221 , piece_id=CLVZGYZLSXB3J2JULQLJZLJZ5LYUYTIPAFFADSRPTLJR6GLYBCHA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.448201499 (71h59m37.319284534s) [noise=true , address=145.79.227.27:4690 , piece_id=TEPT2OC5P7XBKKLQWVDLL6ISOAR4O7GKTZR25TKC5XO6WFF67SBQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.468412445 (71h59m37.33949548s) [noise=true , address=145.79.227.23:4545 , piece_id=YVHM5CAIVDBCYNZMSOD4KQHO6NPVMOX47SY7ALX5FBT2FMYFOK4A, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:37.819874228 (71h59m37.690957273s) [noise=true , address=145.79.227.2:4113 , piece_id=FIAB4XSDITJCYE4JQQG7NZVFPK4PE4HNIZPLUHMP4NVR6EDQJKVA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:38.088706208 (71h59m37.959789243s) [noise=true , address=145.79.227.40:4051 , piece_id=5G24WVCPMQAAUSQWUE3NGAQBYCCQEJJS3MPOTJJIOT72U7PCJWTQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: <nil>
Dec 8 15:28:38.088724744 (71h59m37.959807779s) [seg_pos={0 0}, upload=63035]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:28:38.088896090 (71h59m37.959979135s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 8 15:28:38.088902031 (71h59m37.959985066s) [noise=true , address=145.79.227.41:4085 , piece_id=AUGKDVX54HIRNHJ4IIDTKWMHID7QUIWKYXNDBFF6XNO27OGWSQ5A, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089127871 (71h59m37.960210906s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: read tcp 188.166.225.170:37544->145.79.227.16:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:28:38.089134815 (71h59m37.96021785s) [noise=true , address=145.79.227.16:4912 , piece_id=C6W3Q6EHJZJFI6PTRSPQAQW2P6C33VPNMBRA5J5ON2LBSYLAJAIA, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089154402 (71h59m37.960237457s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34412->145.79.227.32:4834: use of closed network connection
Dec 8 15:28:38.089179560 (71h59m37.960262605s) [noise=true , address=145.79.227.32:4834 , piece_id=B7UCNUTXFRC22DSFBCBJ722U2IEFM3ILDIJPN7LZX3J5DLST6URQ, seg_pos={0 0}, upload=63035]: Long tail cancellation due to optimal threshold
Dec 8 15:28:38.089210268 (71h59m37.960293304s) [seg_pos={0 0}, upload=63035]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:28:38.089327862 (71h59m37.960410907s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:28:38.089335797 (71h59m37.960418832s) [upload=63035]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:28:38.409700686 (71h59m38.280783721s) [upload=63035]: Done uploading segments...
Dec 8 15:28:38.409724240 (71h59m38.280807285s) [upload=63035]: Upload finished. err: <nil>
Duration: 30.003875587s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X] duration:
5.001730211s trace:
https://jaeger.dev.storj.io/trace/567cf271cf4a7926
FAIL (2025-12-08 15:28:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001772613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
10.222191298s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:28:38 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 10.222215576s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
8.287719733s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:28:26 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.28774969s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X...X.........................................X...........] duration:
5.994617276s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:28:23 UTC)
orbiter warning: context deadline exceeded
Duration: 5.994644117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................X............X.X.XX.XX.XX..] duration:
6.297835051s trace:
https://jaeger.dev.storj.io/trace/4fc0a95d29f0fb5f
FAIL (2025-12-08 15:27:54 UTC)
orbiter warning: context deadline exceeded
Duration: 6.298050582s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.] duration:
3.761560489s trace:
https://jaeger.dev.storj.io/trace/432731bcc019157c
FAIL (2025-12-08 15:27:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.761590795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
14.699433905s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:27:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 14.699463941s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
8.176592449s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:27:35 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 8.176622966s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
7.495601156s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:27:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.495633037s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X...X.........................................X..........] duration:
5.899168831s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:27:29 UTC)
orbiter warning: context deadline exceeded
Duration: 5.899216021s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.809827301s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:27:16 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 1.809855745s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X...X.........................................X.........] duration:
6.068551556s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:27:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.068581846s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................................X............X.X.XX.XX.XX.] duration:
26.082691343s trace:
https://jaeger.dev.storj.io/trace/4fc0a95d29f0fb5f
FAIL (2025-12-08 15:27:08 UTC)
orbiter warning: context deadline exceeded
Duration: 26.082723944s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
7.6070522s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:26:51 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.60707831s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X] duration:
5.001642581s trace:
https://jaeger.dev.storj.io/trace/432731bcc019157c
FAIL (2025-12-08 15:26:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001666368s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.783376212s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:26:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.783402522s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
7.850838117s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:26:34 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 7.850865079s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
10.517620864s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:26:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.517648887s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
18.714678467s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:26:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 18.714803745s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
21.979840896s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:26:14 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 21.979866066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X...X.........................................X........] duration:
6.257590219s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:26:13 UTC)
orbiter warning: context deadline exceeded
Duration: 6.257630196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.] duration:
3.681350899s trace:
https://jaeger.dev.storj.io/trace/24e9d0c9083c41c5
FAIL (2025-12-08 15:26:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.681380715s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................................X............X.X.XX.XX.XX] duration:
30.00107778s trace:
https://jaeger.dev.storj.io/trace/4fc0a95d29f0fb5f
FAIL (2025-12-08 15:26:00 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001106174s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
15.231325973s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:25:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 15.231346743s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X] duration:
5.001583504s trace:
https://jaeger.dev.storj.io/trace/24e9d0c9083c41c5
FAIL (2025-12-08 15:25:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001606297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.308401393s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:25:44 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.308427494s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...] duration:
5.000798144s trace:
https://jaeger.dev.storj.io/trace/74979700be69e10e
FAIL (2025-12-08 15:25:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000822921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
8.446893481s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:25:20 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.446929959s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X...X.........................................X.......] duration:
9.66328671s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:25:17 UTC)
orbiter warning: context deadline exceeded
Duration: 9.663339171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
18.373914408s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:25:10 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 18.373944647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
9.541242144s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:25:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 9.541256973s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X..] duration:
5.000893506s trace:
https://jaeger.dev.storj.io/trace/74979700be69e10e
FAIL (2025-12-08 15:25:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000909616s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
7.703986463s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:25:04 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.704013304s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................................X............X.X.XX.XX.X] duration:
30.001116713s trace:
https://jaeger.dev.storj.io/trace/4cccd939171a0cef
FAIL (2025-12-08 15:24:47 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001161419s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
3.829686554s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:24:29 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 3.829710872s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
17.629779978s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:24:24 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 17.629806618s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X.] duration:
5.001415228s trace:
https://jaeger.dev.storj.io/trace/74979700be69e10e
FAIL (2025-12-08 15:24:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001433794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
11.425631199s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:24:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 11.425662327s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X...X.........................................X......] duration:
9.689686315s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:24:08 UTC)
orbiter warning: context deadline exceeded
Duration: 9.689716894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
7.690412678s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:23:50 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.690440681s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
13.906278319s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:23:33 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 13.906306461s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........X...X.........................................X.....] duration:
10.428781841s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:23:30 UTC)
orbiter warning: context deadline exceeded
Duration: 10.428804585s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....................................X............X.X.XX.XX.] duration:
9.470553294s trace:
https://jaeger.dev.storj.io/trace/387f109cc156b7bc
FAIL (2025-12-08 15:23:22 UTC)
orbiter warning: context deadline exceeded
Duration: 9.470570136s
edge
gateway
us-select-1
|