Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..XXX.XXX.XXX.....X......X......X........X.X.X..X..X.....XXX] duration:
30.000629539s trace:
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:42:36 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000638676s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X...X.X.X........................................X.......] duration:
9.434786993s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:42:35 UTC)
orbiter warning: context deadline exceeded
Duration: 9.434813222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...XXX.XXX.XXX.....X......X......X........X.X.X..X..X.....XX] duration:
30.001169396s trace:
https://jaeger.dev.storj.io/trace/bc5f91eb4fb5ac9
FAIL (2025-12-30 14:42:06 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001180787s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration:
5.105457363s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:42:02 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.105485467s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X.] duration:
4.467117406s trace:
https://jaeger.dev.storj.io/trace/42550e4c986f7c29
FAIL (2025-12-30 14:41:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.467174746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
28.207692758s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:41:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 28.207710921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [....XXX.XXX.XXX.....X......X......X........X.X.X..X..X.....X] duration:
30.001249016s trace:
https://jaeger.dev.storj.io/trace/791411a24e895af
FAIL (2025-12-30 14:41:36 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001272351s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X....X....X.....X...X....X......X.......X..X....X..X...X..XX] duration:
30.002523995s trace:
https://jaeger.dev.storj.io/trace/58ae2cef750db2b8
FAIL (2025-12-30 14:41:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 14:41:03.708649647 (599h12m3.579732692s) [upload=525200]: Starting upload
Dec 30 14:41:03.708743957 (599h12m3.579827002s) [upload=525200]: Done starting upload
Dec 30 14:41:03.709040000 (599h12m3.580123045s) [upload=525200]: Uploading segments...
Dec 30 14:41:03.709131755 (599h12m3.58021479s) [upload=525200]: Got next segment. Inline: false
Dec 30 14:41:03.709183673 (599h12m3.580266719s) [upload=525200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:41:03.709189534 (599h12m3.58027258s) [upload=525200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:41:03.989333019 (599h12m3.860416054s) [seg_pos={0 0}, upload=525200]: Begin upload segment...
Dec 30 14:41:03.989397462 (599h12m3.860480497s) [seg_pos={0 0}, upload=525200]: Done begin upload segment.
Dec 30 14:41:03.989427460 (599h12m3.860510495s) [noise=true , address=77.90.31.15:4700 , piece_id=IX6TVWTITX6K4HVXFF7Z6PJF4WSBM6BL4PALXTHUUZM3GNWCD7SQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.990460818 (599h12m3.861543863s) [noise=true , address=145.79.227.35:4482 , piece_id=4QKMGC5ZYV3OKCK4FDFY3UVVVB25VK24O5W5YBTAGD5G2KUEGKZQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.990884093 (599h12m3.861967138s) [noise=true , address=45.91.116.10:4081 , piece_id=37XQBJ6WIOX2Z5LJ5IPUK6AHICTC4VMHCNSQ7NLZUHZBJAK3AILQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.991253676 (599h12m3.862336712s) [noise=true , address=145.79.227.45:4229 , piece_id=HETSEGK7FJHSPFASRBHGWDQLVGBJWGDV6E3CMEJAVPXMGPY5QQUA, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.991693054 (599h12m3.862776089s) [noise=true , address=77.90.31.21:4880 , piece_id=76X5KVV2DVV5RTYRT7OZMCRHAAX756IDEPCDEHUK65V4DKCLOP3Q, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.992428173 (599h12m3.863511208s) [noise=true , address=145.79.227.23:4545 , piece_id=X7VKFVMMKHOO4N4J224SANRSIB4JB3JB6LX5ZHY3S7RHPPO36TNQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.992732864 (599h12m3.863815909s) [noise=true , address=145.79.227.16:4911 , piece_id=DNP5RP6JP67YC24FSOIAL4ALTBZDIQG7PG3QZQZJJYAKBZX4TRPQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.993079554 (599h12m3.864162599s) [noise=true , address=145.79.227.12:4765 , piece_id=QBRF4JJ4QWQBHXNSUGYQUVCDOFZHY47BN43OFVAJZGKY3FJN2C7Q, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.993280926 (599h12m3.864363972s) [noise=true , address=145.79.227.13:4803 , piece_id=XKX4FA2SE6K7LV3AZLPS4B7UKAQ73IBL5TPJJJLMNMJ2QQZEEJMQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.993521475 (599h12m3.86460451s) [noise=true , address=45.91.116.9:4261 , piece_id=RKKBLP27VDJ5HFKGPI67ABNCYVCSLFXTJGEKORPTVYQSJV2QUVKQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.993923360 (599h12m3.865006395s) [noise=true , address=145.79.227.2:4113 , piece_id=RHAIFCGR5AKRMSSE33CZHKFIDVIC7EWCIPRBAAIO6IDZXLIAVUTQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.994082423 (599h12m3.865165458s) [noise=true , address=77.90.31.14:4662 , piece_id=3LTQX2MHOS5PPFMFJGMDOHYBM4ISRSGQQHCXA4XR4AFTTDDRER3A, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.994286071 (599h12m3.865369146s) [noise=true , address=145.79.227.36:4519 , piece_id=DYHTMJ42ZREHR226ND6D4IVY6QQL2XIK5FSJPZKLOXS3WJEIPPRA, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.994828464 (599h12m3.865911509s) [noise=true , address=145.79.227.7:4295 , piece_id=CA64Y6Q2X4PP6IPPC4WZUDTPZORIAXDH6XM2RE377NDEGLGW5XGQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.994868169 (599h12m3.865951204s) [noise=true , address=145.79.227.25:4617 , piece_id=J2NBK6TUS3XH43ONWXFE7EMXFVM7V5ZEEZIIV2YPKUV2XMQLZJJQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:03.994928404 (599h12m3.866011449s) [noise=true , address=145.79.227.41:4085 , piece_id=I4J63TZFEV2H7PWCMTUOODAWV5XC6NAQJF7HHM6DZDGS2APBKBUQ, seg_pos={0 0}, upload=525200]: Uploading piece...
Dec 30 14:41:05.691030683 (599h12m5.562113728s) [upload=525200]: Next returned nil segment
Dec 30 14:41:05.691047856 (599h12m5.562130901s) [upload=525200]: Waiting for error group managing segments...
Dec 30 14:41:05.971649954 (599h12m5.842732989s) [noise=true , address=145.79.227.16:4911 , piece_id=DNP5RP6JP67YC24FSOIAL4ALTBZDIQG7PG3QZQZJJYAKBZX4TRPQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:05.990069217 (599h12m5.861152282s) [noise=true , address=145.79.227.7:4295 , piece_id=CA64Y6Q2X4PP6IPPC4WZUDTPZORIAXDH6XM2RE377NDEGLGW5XGQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:05.990646916 (599h12m5.861729951s) [noise=true , address=145.79.227.23:4545 , piece_id=X7VKFVMMKHOO4N4J224SANRSIB4JB3JB6LX5ZHY3S7RHPPO36TNQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:05.991214878 (599h12m5.862297923s) [noise=true , address=145.79.227.25:4617 , piece_id=J2NBK6TUS3XH43ONWXFE7EMXFVM7V5ZEEZIIV2YPKUV2XMQLZJJQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:05.997426144 (599h12m5.86850919s) [noise=true , address=145.79.227.2:4113 , piece_id=RHAIFCGR5AKRMSSE33CZHKFIDVIC7EWCIPRBAAIO6IDZXLIAVUTQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:05.998109246 (599h12m5.869192291s) [noise=true , address=145.79.227.45:4229 , piece_id=HETSEGK7FJHSPFASRBHGWDQLVGBJWGDV6E3CMEJAVPXMGPY5QQUA, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.001834138 (599h12m5.872917194s) [noise=true , address=145.79.227.12:4765 , piece_id=QBRF4JJ4QWQBHXNSUGYQUVCDOFZHY47BN43OFVAJZGKY3FJN2C7Q, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.002954472 (599h12m5.874037507s) [noise=true , address=145.79.227.41:4085 , piece_id=I4J63TZFEV2H7PWCMTUOODAWV5XC6NAQJF7HHM6DZDGS2APBKBUQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.009511717 (599h12m5.880594762s) [noise=true , address=145.79.227.36:4519 , piece_id=DYHTMJ42ZREHR226ND6D4IVY6QQL2XIK5FSJPZKLOXS3WJEIPPRA, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.069767905 (599h12m5.94085095s) [noise=true , address=145.79.227.13:4803 , piece_id=XKX4FA2SE6K7LV3AZLPS4B7UKAQ73IBL5TPJJJLMNMJ2QQZEEJMQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.162836801 (599h12m6.033919836s) [noise=true , address=77.90.31.15:4700 , piece_id=IX6TVWTITX6K4HVXFF7Z6PJF4WSBM6BL4PALXTHUUZM3GNWCD7SQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.284007178 (599h12m6.155090213s) [noise=true , address=77.90.31.21:4880 , piece_id=76X5KVV2DVV5RTYRT7OZMCRHAAX756IDEPCDEHUK65V4DKCLOP3Q, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.519138894 (599h12m6.390221939s) [noise=true , address=45.91.116.9:4261 , piece_id=RKKBLP27VDJ5HFKGPI67ABNCYVCSLFXTJGEKORPTVYQSJV2QUVKQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: <nil>
Dec 30 14:41:06.519164012 (599h12m6.390247047s) [seg_pos={0 0}, upload=525200]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:41:06.519393057 (599h12m6.390476102s) [noise=true , address=145.79.227.35:4482 , piece_id=4QKMGC5ZYV3OKCK4FDFY3UVVVB25VK24O5W5YBTAGD5G2KUEGKZQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 14:41:06.519502576 (599h12m6.390585621s) [noise=true , address=145.79.227.35:4482 , piece_id=4QKMGC5ZYV3OKCK4FDFY3UVVVB25VK24O5W5YBTAGD5G2KUEGKZQ, seg_pos={0 0}, upload=525200]: Long tail cancellation due to optimal threshold
Dec 30 14:41:06.519508266 (599h12m6.390591311s) [noise=true , address=45.91.116.10:4081 , piece_id=37XQBJ6WIOX2Z5LJ5IPUK6AHICTC4VMHCNSQ7NLZUHZBJAK3AILQ, seg_pos={0 0}, upload=525200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 30 14:41:06.519516913 (599h12m6.390599949s) [noise=true , address=45.91.116.10:4081 , piece_id=37XQBJ6WIOX2Z5LJ5IPUK6AHICTC4VMHCNSQ7NLZUHZBJAK3AILQ, seg_pos={0 0}, upload=525200]: Long tail cancellation due to optimal threshold
Dec 30 14:41:06.519757061 (599h12m6.390840096s) [noise=true , address=77.90.31.14:4662 , piece_id=3LTQX2MHOS5PPFMFJGMDOHYBM4ISRSGQQHCXA4XR4AFTTDDRER3A, seg_pos={0 0}, upload=525200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Dec 30 14:41:06.519766749 (599h12m6.390849794s) [noise=true , address=77.90.31.14:4662 , piece_id=3LTQX2MHOS5PPFMFJGMDOHYBM4ISRSGQQHCXA4XR4AFTTDDRER3A, seg_pos={0 0}, upload=525200]: Long tail cancellation due to optimal threshold
Dec 30 14:41:06.519787378 (599h12m6.390870423s) [seg_pos={0 0}, upload=525200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:41:06.519866219 (599h12m6.390949264s) [upload=525200]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:41:06.519874254 (599h12m6.390957289s) [upload=525200]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:41:06.810953140 (599h12m6.682036175s) [upload=525200]: Done uploading segments...
Dec 30 14:41:06.810976595 (599h12m6.68205964s) [upload=525200]: Upload finished. err: <nil>
Duration: 30.002546738s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....X...X.X.X........................................X......] duration:
9.40396505s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:41:31 UTC)
orbiter warning: context deadline exceeded
Duration: 9.403978917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....X...X.X.X........................................X.....] duration:
29.322287362s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:41:21 UTC)
orbiter warning: context deadline exceeded
Duration: 29.322311749s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
15.976226214s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:41:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 15.976252603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...X...................................................X....] duration:
8.898901341s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:40:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.898951075s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X....X....X.....X...X....X......X.......X..X....X..X...X..X] duration:
30.004096706s trace:
https://jaeger.dev.storj.io/trace/6c6702f3e03157de
FAIL (2025-12-30 14:40:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: HJMLPV57Q7ULJSRNX5QJLPENTNFIFOE5WAGLRUINCOLCGBRJVNKQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: KVOTESGNR2S2D7T2EZG2L4MTXUO5QZGHARR3NZ443BGQCTTAPJ7A) context deadline exceeded
======LOGS======
Dec 30 14:40:23.756295783 (599h11m23.627378818s) [upload=525190]: Starting upload
Dec 30 14:40:23.756366478 (599h11m23.627449523s) [upload=525190]: Done starting upload
Dec 30 14:40:23.756444516 (599h11m23.627527561s) [upload=525190]: Uploading segments...
Dec 30 14:40:23.756628397 (599h11m23.627711442s) [upload=525190]: Got next segment. Inline: false
Dec 30 14:40:23.756680726 (599h11m23.627763761s) [upload=525190]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:40:23.756686687 (599h11m23.627769722s) [upload=525190]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:40:24.004651420 (599h11m23.875734455s) [seg_pos={0 0}, upload=525190]: Begin upload segment...
Dec 30 14:40:24.004706936 (599h11m23.875789981s) [seg_pos={0 0}, upload=525190]: Done begin upload segment.
Dec 30 14:40:24.004741692 (599h11m23.875824737s) [noise=true , address=145.79.227.9:4366 , piece_id=SRU72BJ63VR5V47L7S7WTCISEWUHMPQJEVLCHBIN5VLM3T5VLJ7A, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.004796846 (599h11m23.875879891s) [noise=true , address=145.79.227.21:4474 , piece_id=HYOQ5IEIODEENT5FSKWDKTS36PM2IO7EO3MCMUWFYV5FEMMMQS3A, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.005569227 (599h11m23.876652262s) [noise=true , address=145.79.227.23:4547 , piece_id=7X52OGQKLAEFJBZHQI5DTGEZJRM7O4LU4DXBUWB7LZCIX74WDVTA, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.005845774 (599h11m23.876928819s) [noise=true , address=145.79.227.2:4115 , piece_id=OCNTMK3JAUXWYJTU2IGM5ZCNAFRLRL42UQJ2QQMGWNHGL3VLHDWA, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.005991691 (599h11m23.877074726s) [noise=true , address=145.79.227.25:4617 , piece_id=G53HH7FIDKF75F74SDEWNGZURUSYRPPZPRENCX2M2PP4KMDIZSCQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.006378147 (599h11m23.877461182s) [noise=true , address=145.79.227.32:4836 , piece_id=PI7WQHVMZ5FBOQPANPIREHUSHXM442OKLKTBOQIPVQJZACANY7YQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.006519035 (599h11m23.87760208s) [noise=true , address=145.79.227.15:4875 , piece_id=VV2FKYXS3NSSB24TGI64VBSJCJ7RJVHEU6BI5HMUSXSIOBTRTCYQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.006715580 (599h11m23.877798615s) [noise=true , address=45.91.116.12:4227 , piece_id=KVOTESGNR2S2D7T2EZG2L4MTXUO5QZGHARR3NZ443BGQCTTAPJ7A, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.007053333 (599h11m23.878136368s) [noise=true , address=145.79.227.5:4223 , piece_id=6UVGK5X2CED72S7TF3MY6KBQ2H7DYZIYF4XDZNPDB7W376AQJ7OA, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.007163202 (599h11m23.878246237s) [noise=true , address=145.79.227.18:4006 , piece_id=BHLRALFHPAAMWQTTRACBCMRTWNAE22YFQ3OEBBWCGDWQU4KVIHFQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.007611957 (599h11m23.878695002s) [noise=true , address=145.79.227.13:4802 , piece_id=2FNKKKN5HCISKWRH6SW24QNDIKOMR5YR7CF2D5TEZKXZ5MJFZRMQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.007927688 (599h11m23.879010733s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XZ2XIKFGWY5BD2VJZ2LIMX62F7QSCJQUHGVBJQVLQCABH5M53Q, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.008083444 (599h11m23.879166479s) [noise=true , address=45.91.116.8:4045 , piece_id=JYB6PAI5GRANGM2T2DD4ELSVL2KOTZFX6JZG77U5OMN6GE55TSAQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.008542248 (599h11m23.879625293s) [noise=true , address=77.90.31.18:4805 , piece_id=HJMLPV57Q7ULJSRNX5QJLPENTNFIFOE5WAGLRUINCOLCGBRJVNKQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.008621189 (599h11m23.879704234s) [noise=true , address=145.79.227.33:4871 , piece_id=7S2WOKSPNYVQCZARVZP6EMPJ7JX4FZIQ6KNNEIGF3KL4KNSE3DBQ, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:24.009778903 (599h11m23.880861948s) [noise=true , address=145.79.227.39:4014 , piece_id=VA4GQJ4S2DS42SLUM2MGAAIWYC5FDXON2QUW3PQQY4MNRH4OXSXA, seg_pos={0 0}, upload=525190]: Uploading piece...
Dec 30 14:40:25.709288329 (599h11m25.580371374s) [upload=525190]: Next returned nil segment
Dec 30 14:40:25.709317134 (599h11m25.580400169s) [upload=525190]: Waiting for error group managing segments...
Dec 30 14:40:25.990736929 (599h11m25.861819964s) [noise=true , address=145.79.227.18:4006 , piece_id=BHLRALFHPAAMWQTTRACBCMRTWNAE22YFQ3OEBBWCGDWQU4KVIHFQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:25.998373230 (599h11m25.869456265s) [noise=true , address=145.79.227.13:4802 , piece_id=2FNKKKN5HCISKWRH6SW24QNDIKOMR5YR7CF2D5TEZKXZ5MJFZRMQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.025267299 (599h11m25.896350345s) [noise=true , address=145.79.227.5:4223 , piece_id=6UVGK5X2CED72S7TF3MY6KBQ2H7DYZIYF4XDZNPDB7W376AQJ7OA, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.026550273 (599h11m25.897633318s) [noise=true , address=145.79.227.32:4836 , piece_id=PI7WQHVMZ5FBOQPANPIREHUSHXM442OKLKTBOQIPVQJZACANY7YQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.027103326 (599h11m25.898186371s) [noise=true , address=145.79.227.21:4474 , piece_id=HYOQ5IEIODEENT5FSKWDKTS36PM2IO7EO3MCMUWFYV5FEMMMQS3A, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.029395851 (599h11m25.900478896s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XZ2XIKFGWY5BD2VJZ2LIMX62F7QSCJQUHGVBJQVLQCABH5M53Q, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.083353044 (599h11m25.954436089s) [noise=true , address=145.79.227.2:4115 , piece_id=OCNTMK3JAUXWYJTU2IGM5ZCNAFRLRL42UQJ2QQMGWNHGL3VLHDWA, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.090051970 (599h11m25.961135005s) [noise=true , address=145.79.227.25:4617 , piece_id=G53HH7FIDKF75F74SDEWNGZURUSYRPPZPRENCX2M2PP4KMDIZSCQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.100389063 (599h11m25.971472098s) [noise=true , address=145.79.227.9:4366 , piece_id=SRU72BJ63VR5V47L7S7WTCISEWUHMPQJEVLCHBIN5VLM3T5VLJ7A, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.111228453 (599h11m25.982311488s) [noise=true , address=145.79.227.23:4547 , piece_id=7X52OGQKLAEFJBZHQI5DTGEZJRM7O4LU4DXBUWB7LZCIX74WDVTA, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.176417773 (599h11m26.047500818s) [noise=true , address=145.79.227.39:4014 , piece_id=VA4GQJ4S2DS42SLUM2MGAAIWYC5FDXON2QUW3PQQY4MNRH4OXSXA, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:26.880409428 (599h11m26.751492473s) [noise=true , address=77.90.31.18:4805 , piece_id=HJMLPV57Q7ULJSRNX5QJLPENTNFIFOE5WAGLRUINCOLCGBRJVNKQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:27.112744077 (599h11m26.983827122s) [noise=true , address=45.91.116.12:4227 , piece_id=KVOTESGNR2S2D7T2EZG2L4MTXUO5QZGHARR3NZ443BGQCTTAPJ7A, seg_pos={0 0}, upload=525190]: Done uploading piece. err: <nil>
Dec 30 14:40:27.112765147 (599h11m26.983848192s) [seg_pos={0 0}, upload=525190]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:40:27.112962763 (599h11m26.984045809s) [noise=true , address=45.91.116.8:4045 , piece_id=JYB6PAI5GRANGM2T2DD4ELSVL2KOTZFX6JZG77U5OMN6GE55TSAQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 30 14:40:27.112969406 (599h11m26.984052441s) [noise=true , address=45.91.116.8:4045 , piece_id=JYB6PAI5GRANGM2T2DD4ELSVL2KOTZFX6JZG77U5OMN6GE55TSAQ, seg_pos={0 0}, upload=525190]: Long tail cancellation due to optimal threshold
Dec 30 14:40:27.113083043 (599h11m26.984166108s) [noise=true , address=145.79.227.15:4875 , piece_id=VV2FKYXS3NSSB24TGI64VBSJCJ7RJVHEU6BI5HMUSXSIOBTRTCYQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF
Dec 30 14:40:27.113118661 (599h11m26.984201716s) [noise=true , address=145.79.227.15:4875 , piece_id=VV2FKYXS3NSSB24TGI64VBSJCJ7RJVHEU6BI5HMUSXSIOBTRTCYQ, seg_pos={0 0}, upload=525190]: Long tail cancellation due to optimal threshold
Dec 30 14:40:27.113272173 (599h11m26.984355208s) [noise=true , address=145.79.227.33:4871 , piece_id=7S2WOKSPNYVQCZARVZP6EMPJ7JX4FZIQ6KNNEIGF3KL4KNSE3DBQ, seg_pos={0 0}, upload=525190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 30 14:40:27.113279667 (599h11m26.984362702s) [noise=true , address=145.79.227.33:4871 , piece_id=7S2WOKSPNYVQCZARVZP6EMPJ7JX4FZIQ6KNNEIGF3KL4KNSE3DBQ, seg_pos={0 0}, upload=525190]: Long tail cancellation due to optimal threshold
Dec 30 14:40:27.113308211 (599h11m26.984391246s) [seg_pos={0 0}, upload=525190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:40:27.113384617 (599h11m26.984467652s) [upload=525190]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:40:27.113389195 (599h11m26.98447224s) [upload=525190]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:40:27.414498382 (599h11m27.285581427s) [upload=525190]: Done uploading segments...
Dec 30 14:40:27.414526785 (599h11m27.285609831s) [upload=525190]: Upload finished. err: <nil>
Duration: 30.004120912s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
26.383335922s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:40:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 26.383363234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X] duration:
5.001559184s trace:
https://jaeger.dev.storj.io/trace/42550e4c986f7c29
FAIL (2025-12-30 14:40:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001584252s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.] duration:
4.534767307s trace:
https://jaeger.dev.storj.io/trace/753cb210fde717d5
FAIL (2025-12-30 14:40:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.534818564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [X.....X...X.X.X........................................X....] duration:
13.034777867s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:40:17 UTC)
orbiter warning: context deadline exceeded
Duration: 13.034815328s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X....XXX.XXX.XXX.....X......X......X........X.X.X..X..X.....] duration:
18.175016888s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:40:16 UTC)
orbiter warning: context deadline exceeded
Duration: 18.175043489s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [....X...................................................X...] duration:
8.784764186s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:40:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.784786929s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX....XXX.XXX.XXX.....X......X......X........X.X.X..X..X....] duration:
19.707739613s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:39:56 UTC)
orbiter warning: context deadline exceeded
Duration: 19.707764179s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX] duration:
5.00150288s trace:
https://jaeger.dev.storj.io/trace/753cb210fde717d5
FAIL (2025-12-30 14:39:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001520063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.X.....X...X.X.X........................................X...] duration:
5.784463149s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:39:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.784483227s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..X....X....X.....X...X....X......X.......X..X....X..X...X..] duration:
5.991783348s trace:
https://jaeger.dev.storj.io/trace/4e9df9f6f3140c43
FAIL (2025-12-30 14:39:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ) context deadline exceeded
======LOGS======
Dec 30 14:38:30.982065061 (599h9m30.853148096s) [upload=525162]: Starting upload
Dec 30 14:38:30.982141406 (599h9m30.853224441s) [upload=525162]: Done starting upload
Dec 30 14:38:30.982240375 (599h9m30.85332344s) [upload=525162]: Uploading segments...
Dec 30 14:38:30.982410469 (599h9m30.853493514s) [upload=525162]: Got next segment. Inline: false
Dec 30 14:38:30.982452248 (599h9m30.853535293s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:38:30.982459061 (599h9m30.853542106s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:38:31.227198921 (599h9m31.098281966s) [seg_pos={0 0}, upload=525162]: Begin upload segment...
Dec 30 14:38:31.227263985 (599h9m31.09834703s) [seg_pos={0 0}, upload=525162]: Done begin upload segment.
Dec 30 14:38:31.227313639 (599h9m31.098396684s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.227529711 (599h9m31.098612746s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228032198 (599h9m31.099115233s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228556847 (599h9m31.099639882s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228880503 (599h9m31.099963538s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228975384 (599h9m31.100058429s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229345379 (599h9m31.100428414s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229391346 (599h9m31.100474391s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229764998 (599h9m31.100848043s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230221958 (599h9m31.101305003s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230565041 (599h9m31.101648086s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231196072 (599h9m31.102279108s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231641421 (599h9m31.102724456s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.233048209 (599h9m31.104131254s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235148289 (599h9m31.106231334s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235333200 (599h9m31.106416236s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:32.935561257 (599h9m32.806644292s) [upload=525162]: Next returned nil segment
Dec 30 14:38:32.935576076 (599h9m32.806659121s) [upload=525162]: Waiting for error group managing segments...
Dec 30 14:38:33.226521819 (599h9m33.097604864s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231181672 (599h9m33.102264707s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231956548 (599h9m33.103039583s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.238947359 (599h9m33.110030394s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246426962 (599h9m33.117510007s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246940780 (599h9m33.118023825s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.247203470 (599h9m33.118286505s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.299968713 (599h9m33.171051748s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.380553819 (599h9m33.251636854s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.415327833 (599h9m33.286410879s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.521195682 (599h9m33.392278727s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.527845554 (599h9m33.398928589s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605450165 (599h9m33.47653321s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605468280 (599h9m33.476551315s) [seg_pos={0 0}, upload=525162]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:38:33.605654704 (599h9m33.476737739s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605660365 (599h9m33.47674341s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.605957060 (599h9m33.477040105s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605974473 (599h9m33.477057518s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606009059 (599h9m33.477092104s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.606017204 (599h9m33.477100239s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606056199 (599h9m33.477139244s) [seg_pos={0 0}, upload=525162]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:38:33.606133085 (599h9m33.47721612s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:38:33.606137985 (599h9m33.47722102s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:38:33.893567653 (599h9m33.764650688s) [upload=525162]: Done uploading segments...
Dec 30 14:38:33.893586108 (599h9m33.764669143s) [upload=525162]: Upload finished. err: <nil>
Duration: 5.991807134s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
35.483226544s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:39:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 35.483253996s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X....X....X.....X...X....X......X.......X..X....X..X...X.] duration:
5.78079114s trace:
https://jaeger.dev.storj.io/trace/4e9df9f6f3140c43
FAIL (2025-12-30 14:39:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ) context deadline exceeded
======LOGS======
Dec 30 14:38:30.982065061 (599h9m30.853148096s) [upload=525162]: Starting upload
Dec 30 14:38:30.982141406 (599h9m30.853224441s) [upload=525162]: Done starting upload
Dec 30 14:38:30.982240375 (599h9m30.85332344s) [upload=525162]: Uploading segments...
Dec 30 14:38:30.982410469 (599h9m30.853493514s) [upload=525162]: Got next segment. Inline: false
Dec 30 14:38:30.982452248 (599h9m30.853535293s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:38:30.982459061 (599h9m30.853542106s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:38:31.227198921 (599h9m31.098281966s) [seg_pos={0 0}, upload=525162]: Begin upload segment...
Dec 30 14:38:31.227263985 (599h9m31.09834703s) [seg_pos={0 0}, upload=525162]: Done begin upload segment.
Dec 30 14:38:31.227313639 (599h9m31.098396684s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.227529711 (599h9m31.098612746s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228032198 (599h9m31.099115233s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228556847 (599h9m31.099639882s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228880503 (599h9m31.099963538s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228975384 (599h9m31.100058429s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229345379 (599h9m31.100428414s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229391346 (599h9m31.100474391s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229764998 (599h9m31.100848043s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230221958 (599h9m31.101305003s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230565041 (599h9m31.101648086s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231196072 (599h9m31.102279108s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231641421 (599h9m31.102724456s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.233048209 (599h9m31.104131254s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235148289 (599h9m31.106231334s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235333200 (599h9m31.106416236s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:32.935561257 (599h9m32.806644292s) [upload=525162]: Next returned nil segment
Dec 30 14:38:32.935576076 (599h9m32.806659121s) [upload=525162]: Waiting for error group managing segments...
Dec 30 14:38:33.226521819 (599h9m33.097604864s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231181672 (599h9m33.102264707s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231956548 (599h9m33.103039583s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.238947359 (599h9m33.110030394s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246426962 (599h9m33.117510007s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246940780 (599h9m33.118023825s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.247203470 (599h9m33.118286505s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.299968713 (599h9m33.171051748s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.380553819 (599h9m33.251636854s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.415327833 (599h9m33.286410879s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.521195682 (599h9m33.392278727s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.527845554 (599h9m33.398928589s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605450165 (599h9m33.47653321s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605468280 (599h9m33.476551315s) [seg_pos={0 0}, upload=525162]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:38:33.605654704 (599h9m33.476737739s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605660365 (599h9m33.47674341s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.605957060 (599h9m33.477040105s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605974473 (599h9m33.477057518s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606009059 (599h9m33.477092104s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.606017204 (599h9m33.477100239s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606056199 (599h9m33.477139244s) [seg_pos={0 0}, upload=525162]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:38:33.606133085 (599h9m33.47721612s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:38:33.606137985 (599h9m33.47722102s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:38:33.893567653 (599h9m33.764650688s) [upload=525162]: Done uploading segments...
Dec 30 14:38:33.893586108 (599h9m33.764669143s) [upload=525162]: Upload finished. err: <nil>
Duration: 5.780819605s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..X] duration:
5.000276615s trace:
https://jaeger.dev.storj.io/trace/3bb0a2fa65fe13bd
FAIL (2025-12-30 14:39:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000291684s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..X.....X...X.X.X........................................X..] duration:
6.410609341s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:39:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.410682481s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration:
4.795320983s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:39:07 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 4.795386438s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [....X....X....X.....X...X....X......X.......X..X....X..X...X] duration:
30.003720039s trace:
https://jaeger.dev.storj.io/trace/4e9df9f6f3140c43
FAIL (2025-12-30 14:39:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ) context deadline exceeded
======LOGS======
Dec 30 14:38:30.982065061 (599h9m30.853148096s) [upload=525162]: Starting upload
Dec 30 14:38:30.982141406 (599h9m30.853224441s) [upload=525162]: Done starting upload
Dec 30 14:38:30.982240375 (599h9m30.85332344s) [upload=525162]: Uploading segments...
Dec 30 14:38:30.982410469 (599h9m30.853493514s) [upload=525162]: Got next segment. Inline: false
Dec 30 14:38:30.982452248 (599h9m30.853535293s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:38:30.982459061 (599h9m30.853542106s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:38:31.227198921 (599h9m31.098281966s) [seg_pos={0 0}, upload=525162]: Begin upload segment...
Dec 30 14:38:31.227263985 (599h9m31.09834703s) [seg_pos={0 0}, upload=525162]: Done begin upload segment.
Dec 30 14:38:31.227313639 (599h9m31.098396684s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.227529711 (599h9m31.098612746s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228032198 (599h9m31.099115233s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228556847 (599h9m31.099639882s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228880503 (599h9m31.099963538s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.228975384 (599h9m31.100058429s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229345379 (599h9m31.100428414s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229391346 (599h9m31.100474391s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.229764998 (599h9m31.100848043s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230221958 (599h9m31.101305003s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.230565041 (599h9m31.101648086s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231196072 (599h9m31.102279108s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.231641421 (599h9m31.102724456s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.233048209 (599h9m31.104131254s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235148289 (599h9m31.106231334s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:31.235333200 (599h9m31.106416236s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Uploading piece...
Dec 30 14:38:32.935561257 (599h9m32.806644292s) [upload=525162]: Next returned nil segment
Dec 30 14:38:32.935576076 (599h9m32.806659121s) [upload=525162]: Waiting for error group managing segments...
Dec 30 14:38:33.226521819 (599h9m33.097604864s) [noise=true , address=145.79.227.26:4654 , piece_id=EI3Y5CIUIWDMLZOCPL4ABNUREHFSSM4A2HSHAEDYKXYNVDCXPRXQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231181672 (599h9m33.102264707s) [noise=true , address=145.79.227.23:4546 , piece_id=MDE5XP55KFPDAEJYTYW2546BOZKOJUUUYDFW7MWUYTGYXEOSQCJQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.231956548 (599h9m33.103039583s) [noise=true , address=145.79.227.47:4266 , piece_id=KHEJYLVKRWJJD24AH72O2JUXLRJEVOEZ7YIBHWPIUQ23PUREI6ZQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.238947359 (599h9m33.110030394s) [noise=true , address=145.79.227.20:4439 , piece_id=CLCRNHTWY6VMTAXMTN2OXGXRSHQ3WLBRU2NRITQVQMWYASSVLVAA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246426962 (599h9m33.117510007s) [noise=true , address=145.79.227.42:4122 , piece_id=AHWLOFKQTJHA3LZVG2IARV6RXUMOEVIR7FH3BWHYVCFV56WIZDQA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.246940780 (599h9m33.118023825s) [noise=true , address=145.79.227.15:4876 , piece_id=EJPIKA5B4SJCUJXDCEL6PPXRFUU2YLEPHCNW3FDPON6HNFRS6IRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.247203470 (599h9m33.118286505s) [noise=true , address=145.79.227.45:4231 , piece_id=QZSFMJQ7JE4ANWE6NDZBR74WYXZJD6IDIAIPUOUKKKKR6ZRBAGRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.299968713 (599h9m33.171051748s) [noise=true , address=145.79.227.24:4581 , piece_id=RSMPHOLNEBNYWIWIAM5QHUSNFP6UVTJ3YAO43AZFDABCTZP3RYFA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.380553819 (599h9m33.251636854s) [noise=true , address=77.90.31.13:4629 , piece_id=Z7X6XRW5YXY55MMEQUWLR5GL5LHJKEKVGY5CYMCSA76HN27EGWWA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.415327833 (599h9m33.286410879s) [noise=true , address=77.90.31.11:4554 , piece_id=N2NGTYLAOPTPQFJS6N3DSTCFQ6VLXELK6JFM7A2UO6F6NRWPQQRQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.521195682 (599h9m33.392278727s) [noise=true , address=45.91.116.8:4045 , piece_id=6HNNHKQCJ4QRXYX6IPJX7W5FYOXGDB4LW5PHHY7GNIT5LXJJVMTQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.527845554 (599h9m33.398928589s) [noise=true , address=77.90.31.10:4519 , piece_id=DPY6YCT7CBSL5V2ZYT7ENMHI73RU5F4GDZHLHFBKMHWD4BG3774Q, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605450165 (599h9m33.47653321s) [noise=true , address=77.90.31.12:4590 , piece_id=DE6QDGYM65VWYD3RL5B4VQUSFTGXH2S3ZXA6CEOECZI6OL64WLNQ, seg_pos={0 0}, upload=525162]: Done uploading piece. err: <nil>
Dec 30 14:38:33.605468280 (599h9m33.476551315s) [seg_pos={0 0}, upload=525162]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:38:33.605654704 (599h9m33.476737739s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605660365 (599h9m33.47674341s) [noise=true , address=45.91.116.7:4012 , piece_id=GVDE2ROZBAFRJPJEAIKGP75SEVUR5APN6XDFAJYNIFSWSLPXKYTA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.605957060 (599h9m33.477040105s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.605974473 (599h9m33.477057518s) [noise=true , address=45.91.116.4:4937 , piece_id=V5ZIIA4KEV7K5C6BB2GLPA4AP6S3FLXHOGR34STU5RSANLLR7VVA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606009059 (599h9m33.477092104s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 14:38:33.606017204 (599h9m33.477100239s) [noise=true , address=45.91.116.5:4298 , piece_id=RPYZALXGICYV7IYWD2UUKGN4JD6TQ2BE5JS4S7ATEZVOTANKKKHA, seg_pos={0 0}, upload=525162]: Long tail cancellation due to optimal threshold
Dec 30 14:38:33.606056199 (599h9m33.477139244s) [seg_pos={0 0}, upload=525162]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:38:33.606133085 (599h9m33.47721612s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:38:33.606137985 (599h9m33.47722102s) [upload=525162]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:38:33.893567653 (599h9m33.764650688s) [upload=525162]: Done uploading segments...
Dec 30 14:38:33.893586108 (599h9m33.764669143s) [upload=525162]: Upload finished. err: <nil>
Duration: 30.003742993s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XX....XXX.XXX.XXX.....X......X......X........X.X.X..X..X...] duration:
15.760904681s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:38:55 UTC)
orbiter warning: context deadline exceeded
Duration: 15.76103488s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [......X...................................................X.] duration:
10.891341322s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:38:52 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.891368224s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
15.472185426s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:38:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 15.472200675s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XX.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..] duration:
4.104501736s trace:
https://jaeger.dev.storj.io/trace/50c835c0e597b363
FAIL (2025-12-30 14:38:27 UTC)
orbiter warning: context deadline exceeded
Duration: 4.104526645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
48.328810554s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:38:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 48.32883008s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......X...................................................X] duration:
10.539861302s trace:
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:38:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.539885327s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..XX....XXX.XXX.XXX.....X......X......X........X.X.X..X..X..] duration:
26.49248836s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:38:10 UTC)
orbiter warning: context deadline exceeded
Duration: 26.492517524s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....X....X....X.....X...X....X......X.......X..X....X..X...] duration:
6.09221143s trace:
https://jaeger.dev.storj.io/trace/63ed2a0f3a8a4a6a
FAIL (2025-12-30 14:38:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA) context deadline exceeded
======LOGS======
Dec 30 14:35:58.163947906 (599h6m58.035030941s) [upload=525123]: Starting upload
Dec 30 14:35:58.164014353 (599h6m58.035097388s) [upload=525123]: Done starting upload
Dec 30 14:35:58.164050051 (599h6m58.035133106s) [upload=525123]: Uploading segments...
Dec 30 14:35:58.164208271 (599h6m58.035291307s) [upload=525123]: Got next segment. Inline: false
Dec 30 14:35:58.164234572 (599h6m58.035317607s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:35:58.164238960 (599h6m58.035321995s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:35:58.415276473 (599h6m58.286359508s) [seg_pos={0 0}, upload=525123]: Begin upload segment...
Dec 30 14:35:58.415399928 (599h6m58.286482963s) [seg_pos={0 0}, upload=525123]: Done begin upload segment.
Dec 30 14:35:58.415440516 (599h6m58.286523551s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.415506811 (599h6m58.286589846s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416014999 (599h6m58.287098034s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416306554 (599h6m58.287389589s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416666390 (599h6m58.287749425s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416995366 (599h6m58.288078401s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417307781 (599h6m58.288390816s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417637419 (599h6m58.288720454s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417992976 (599h6m58.289076011s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.419391609 (599h6m58.290474644s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.420504608 (599h6m58.291587643s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.421558134 (599h6m58.292641179s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.422977247 (599h6m58.294060282s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.424367053 (599h6m58.295450088s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.426000713 (599h6m58.297083748s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.427887717 (599h6m58.298970752s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:36:00.014490984 (599h6m59.885574019s) [upload=525123]: Next returned nil segment
Dec 30 14:36:00.014510510 (599h6m59.885593545s) [upload=525123]: Waiting for error group managing segments...
Dec 30 14:36:00.394597652 (599h7m0.265680697s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.399776484 (599h7m0.27088105s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.410339547 (599h7m0.281422592s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.413428229 (599h7m0.284511264s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.421368288 (599h7m0.292451323s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.440478646 (599h7m0.311561681s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.522850836 (599h7m0.393933871s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.671861925 (599h7m0.54294496s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.683667023 (599h7m0.554750058s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.830507419 (599h7m0.701590454s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.859399824 (599h7m0.730482859s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.448641828 (599h7m1.319724863s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614275799 (599h7m1.485358835s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614297251 (599h7m1.485380286s) [seg_pos={0 0}, upload=525123]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:36:01.614556474 (599h7m1.485639509s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614565140 (599h7m1.485648175s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614585379 (599h7m1.485668424s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614597231 (599h7m1.485680266s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614857698 (599h7m1.485940743s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614867877 (599h7m1.485950922s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614894368 (599h7m1.485977403s) [seg_pos={0 0}, upload=525123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:36:01.614989599 (599h7m1.486072634s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:36:01.614994027 (599h7m1.486077063s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:36:01.905271368 (599h7m1.776354423s) [upload=525123]: Done uploading segments...
Dec 30 14:36:01.905339718 (599h7m1.776422763s) [upload=525123]: Upload finished. err: <nil>
Duration: 6.092238943s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...X.....X...X.X.X........................................X.] duration:
8.714723795s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:38:00 UTC)
orbiter warning: context deadline exceeded
Duration: 8.714743944s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X.] duration:
4.710553114s trace:
https://jaeger.dev.storj.io/trace/50c835c0e597b363
FAIL (2025-12-30 14:37:55 UTC)
orbiter warning: context deadline exceeded
Duration: 4.710581758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....X.....X...X.X.X........................................X] duration:
1m0.001693026s trace:
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:37:51 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001706372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X..XX....XXX.XXX.XXX.....X......X......X........X.X.X..X..X.] duration:
24.946297288s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:37:37 UTC)
orbiter warning: context deadline exceeded
Duration: 24.946327755s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXX.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X] duration:
5.001028231s trace:
https://jaeger.dev.storj.io/trace/50c835c0e597b363
FAIL (2025-12-30 14:37:35 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001041517s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [......X....X....X.....X...X....X......X.......X..X....X..X..] duration:
5.955428309s trace:
https://jaeger.dev.storj.io/trace/63ed2a0f3a8a4a6a
FAIL (2025-12-30 14:37:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA) context deadline exceeded
======LOGS======
Dec 30 14:35:58.163947906 (599h6m58.035030941s) [upload=525123]: Starting upload
Dec 30 14:35:58.164014353 (599h6m58.035097388s) [upload=525123]: Done starting upload
Dec 30 14:35:58.164050051 (599h6m58.035133106s) [upload=525123]: Uploading segments...
Dec 30 14:35:58.164208271 (599h6m58.035291307s) [upload=525123]: Got next segment. Inline: false
Dec 30 14:35:58.164234572 (599h6m58.035317607s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:35:58.164238960 (599h6m58.035321995s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:35:58.415276473 (599h6m58.286359508s) [seg_pos={0 0}, upload=525123]: Begin upload segment...
Dec 30 14:35:58.415399928 (599h6m58.286482963s) [seg_pos={0 0}, upload=525123]: Done begin upload segment.
Dec 30 14:35:58.415440516 (599h6m58.286523551s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.415506811 (599h6m58.286589846s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416014999 (599h6m58.287098034s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416306554 (599h6m58.287389589s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416666390 (599h6m58.287749425s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416995366 (599h6m58.288078401s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417307781 (599h6m58.288390816s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417637419 (599h6m58.288720454s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417992976 (599h6m58.289076011s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.419391609 (599h6m58.290474644s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.420504608 (599h6m58.291587643s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.421558134 (599h6m58.292641179s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.422977247 (599h6m58.294060282s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.424367053 (599h6m58.295450088s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.426000713 (599h6m58.297083748s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.427887717 (599h6m58.298970752s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:36:00.014490984 (599h6m59.885574019s) [upload=525123]: Next returned nil segment
Dec 30 14:36:00.014510510 (599h6m59.885593545s) [upload=525123]: Waiting for error group managing segments...
Dec 30 14:36:00.394597652 (599h7m0.265680697s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.399776484 (599h7m0.27088105s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.410339547 (599h7m0.281422592s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.413428229 (599h7m0.284511264s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.421368288 (599h7m0.292451323s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.440478646 (599h7m0.311561681s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.522850836 (599h7m0.393933871s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.671861925 (599h7m0.54294496s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.683667023 (599h7m0.554750058s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.830507419 (599h7m0.701590454s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.859399824 (599h7m0.730482859s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.448641828 (599h7m1.319724863s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614275799 (599h7m1.485358835s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614297251 (599h7m1.485380286s) [seg_pos={0 0}, upload=525123]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:36:01.614556474 (599h7m1.485639509s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614565140 (599h7m1.485648175s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614585379 (599h7m1.485668424s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614597231 (599h7m1.485680266s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614857698 (599h7m1.485940743s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614867877 (599h7m1.485950922s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614894368 (599h7m1.485977403s) [seg_pos={0 0}, upload=525123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:36:01.614989599 (599h7m1.486072634s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:36:01.614994027 (599h7m1.486077063s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:36:01.905271368 (599h7m1.776354423s) [upload=525123]: Done uploading segments...
Dec 30 14:36:01.905339718 (599h7m1.776422763s) [upload=525123]: Upload finished. err: <nil>
Duration: 5.955457425s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
8.03400895s trace:
https://jaeger.dev.storj.io/trace/7aa21b18ea57a13d
FAIL (2025-12-30 14:37:24 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.034031243s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXX.XX.XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.] duration:
3.925984675s trace:
https://jaeger.dev.storj.io/trace/4edc9f49b3b15b2e
FAIL (2025-12-30 14:37:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.926033487s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [X....X.....X...X.X.X........................................] duration:
35.497717994s trace:
https://jaeger.dev.storj.io/trace/1c21809311de722d
FAIL (2025-12-30 14:36:50 UTC)
orbiter warning: context deadline exceeded
Duration: 35.497760456s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
15.352184596s trace:
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:36:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958 , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016 , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343 , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603 , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873 , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680 , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901 , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878 , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768 , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925 , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208 , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072 , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363 , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174 , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633 , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710 , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575 , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708 , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 15.352205566s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X......X....X....X.....X...X....X......X.......X..X....X..X.] duration:
5.730237457s trace:
https://jaeger.dev.storj.io/trace/63ed2a0f3a8a4a6a
FAIL (2025-12-30 14:36:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA) context deadline exceeded
======LOGS======
Dec 30 14:35:58.163947906 (599h6m58.035030941s) [upload=525123]: Starting upload
Dec 30 14:35:58.164014353 (599h6m58.035097388s) [upload=525123]: Done starting upload
Dec 30 14:35:58.164050051 (599h6m58.035133106s) [upload=525123]: Uploading segments...
Dec 30 14:35:58.164208271 (599h6m58.035291307s) [upload=525123]: Got next segment. Inline: false
Dec 30 14:35:58.164234572 (599h6m58.035317607s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:35:58.164238960 (599h6m58.035321995s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:35:58.415276473 (599h6m58.286359508s) [seg_pos={0 0}, upload=525123]: Begin upload segment...
Dec 30 14:35:58.415399928 (599h6m58.286482963s) [seg_pos={0 0}, upload=525123]: Done begin upload segment.
Dec 30 14:35:58.415440516 (599h6m58.286523551s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.415506811 (599h6m58.286589846s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416014999 (599h6m58.287098034s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416306554 (599h6m58.287389589s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416666390 (599h6m58.287749425s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416995366 (599h6m58.288078401s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417307781 (599h6m58.288390816s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417637419 (599h6m58.288720454s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417992976 (599h6m58.289076011s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.419391609 (599h6m58.290474644s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.420504608 (599h6m58.291587643s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.421558134 (599h6m58.292641179s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.422977247 (599h6m58.294060282s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.424367053 (599h6m58.295450088s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.426000713 (599h6m58.297083748s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.427887717 (599h6m58.298970752s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:36:00.014490984 (599h6m59.885574019s) [upload=525123]: Next returned nil segment
Dec 30 14:36:00.014510510 (599h6m59.885593545s) [upload=525123]: Waiting for error group managing segments...
Dec 30 14:36:00.394597652 (599h7m0.265680697s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.399776484 (599h7m0.27088105s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.410339547 (599h7m0.281422592s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.413428229 (599h7m0.284511264s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.421368288 (599h7m0.292451323s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.440478646 (599h7m0.311561681s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.522850836 (599h7m0.393933871s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.671861925 (599h7m0.54294496s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.683667023 (599h7m0.554750058s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.830507419 (599h7m0.701590454s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.859399824 (599h7m0.730482859s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.448641828 (599h7m1.319724863s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614275799 (599h7m1.485358835s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614297251 (599h7m1.485380286s) [seg_pos={0 0}, upload=525123]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:36:01.614556474 (599h7m1.485639509s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614565140 (599h7m1.485648175s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614585379 (599h7m1.485668424s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614597231 (599h7m1.485680266s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614857698 (599h7m1.485940743s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614867877 (599h7m1.485950922s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614894368 (599h7m1.485977403s) [seg_pos={0 0}, upload=525123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:36:01.614989599 (599h7m1.486072634s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:36:01.614994027 (599h7m1.486077063s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:36:01.905271368 (599h7m1.776354423s) [upload=525123]: Done uploading segments...
Dec 30 14:36:01.905339718 (599h7m1.776422763s) [upload=525123]: Upload finished. err: <nil>
Duration: 5.730260332s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X..XX....XXX.XXX.XXX.....X......X......X........X.X.X..X..X] duration:
30.001172263s trace:
https://jaeger.dev.storj.io/trace/11cc10fe700c4053
FAIL (2025-12-30 14:36:32 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001183364s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X......X....X....X.....X...X....X......X.......X..X....X..X] duration:
30.002689825s trace:
https://jaeger.dev.storj.io/trace/63ed2a0f3a8a4a6a
FAIL (2025-12-30 14:36:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA) context deadline exceeded
======LOGS======
Dec 30 14:35:58.163947906 (599h6m58.035030941s) [upload=525123]: Starting upload
Dec 30 14:35:58.164014353 (599h6m58.035097388s) [upload=525123]: Done starting upload
Dec 30 14:35:58.164050051 (599h6m58.035133106s) [upload=525123]: Uploading segments...
Dec 30 14:35:58.164208271 (599h6m58.035291307s) [upload=525123]: Got next segment. Inline: false
Dec 30 14:35:58.164234572 (599h6m58.035317607s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:35:58.164238960 (599h6m58.035321995s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:35:58.415276473 (599h6m58.286359508s) [seg_pos={0 0}, upload=525123]: Begin upload segment...
Dec 30 14:35:58.415399928 (599h6m58.286482963s) [seg_pos={0 0}, upload=525123]: Done begin upload segment.
Dec 30 14:35:58.415440516 (599h6m58.286523551s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.415506811 (599h6m58.286589846s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416014999 (599h6m58.287098034s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416306554 (599h6m58.287389589s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416666390 (599h6m58.287749425s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.416995366 (599h6m58.288078401s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417307781 (599h6m58.288390816s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417637419 (599h6m58.288720454s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.417992976 (599h6m58.289076011s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.419391609 (599h6m58.290474644s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.420504608 (599h6m58.291587643s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.421558134 (599h6m58.292641179s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.422977247 (599h6m58.294060282s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.424367053 (599h6m58.295450088s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.426000713 (599h6m58.297083748s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:35:58.427887717 (599h6m58.298970752s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Uploading piece...
Dec 30 14:36:00.014490984 (599h6m59.885574019s) [upload=525123]: Next returned nil segment
Dec 30 14:36:00.014510510 (599h6m59.885593545s) [upload=525123]: Waiting for error group managing segments...
Dec 30 14:36:00.394597652 (599h7m0.265680697s) [noise=true , address=145.79.227.6:4257 , piece_id=WHWUSY4B6HVXPDKIMLA4QU7TDW4LGVFMYS3TTJID4WND52OHTO2A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.399776484 (599h7m0.27088105s) [noise=true , address=145.79.227.46:4601 , piece_id=663U3Z2TUNUDJ7DCAM25S4PNLU64JFAEP24FGJZYLCMJW4MRVTZQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.410339547 (599h7m0.281422592s) [noise=true , address=145.79.227.11:4730 , piece_id=OHALB3XRROT4DZXGYVUTGD7FWQ5FURJC7SFBZHROWSHSJXB7JAFQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.413428229 (599h7m0.284511264s) [noise=true , address=145.79.227.21:4475 , piece_id=RWOYLXL7LU43LWQBQGDEQIRBFK7M3CK3L22H7WXFJVZCE3YTBGTA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.421368288 (599h7m0.292451323s) [noise=true , address=145.79.227.24:4581 , piece_id=MKDOYOQAUI32TRJWT2CNYQMUVVG67YFATCJT6CZQ5DA4MB32YAIA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.440478646 (599h7m0.311561681s) [noise=true , address=145.79.227.17:4971 , piece_id=XBKBJNQTZDIF4SDL2QIJHUMR5WDCHOWS2TJD72CFNN763GOXR4TQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.522850836 (599h7m0.393933871s) [noise=true , address=145.79.227.5:4221 , piece_id=FZNQ5W55HUOLH4YJN4H2AUYDLCTDNBQHGC7X6R6GN4DHKT2B7G7A, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.671861925 (599h7m0.54294496s) [noise=true , address=77.90.31.13:4626 , piece_id=C7PL4HCOI6A2HQINCJB4SADVL6E3KQHFMSHMI6LGWJZIQYCXSTAA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.683667023 (599h7m0.554750058s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDREMWWLPWQHH2HGIZZS4V5YQBPH24HKJ7X2EK2OCBEL63LWDRQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.830507419 (599h7m0.701590454s) [noise=true , address=45.91.116.10:4084 , piece_id=YIXXPYGLSXBF5MZ4WYQXZGPP6A6NOUWAB6CRPJPTNVBNMVX3NCNA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:00.859399824 (599h7m0.730482859s) [noise=true , address=45.91.116.8:4045 , piece_id=4G7KBRW5472EEL6T43NRPUEVYNAPDG7JVD6ZV5MGZ2IGPIEWZMIQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.448641828 (599h7m1.319724863s) [noise=true , address=45.91.116.6:4973 , piece_id=4MGJJEO24IN76LJ4BVWOQO4C62APJFNFPTRUHDA4F7EOO27OG3QA, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614275799 (599h7m1.485358835s) [noise=true , address=45.91.116.9:4262 , piece_id=E7DDR7YGHAPFJYNCK4RHAULBNE5IVIZGQZW7OC26OUTKXIG5AGEQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: <nil>
Dec 30 14:36:01.614297251 (599h7m1.485380286s) [seg_pos={0 0}, upload=525123]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:36:01.614556474 (599h7m1.485639509s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614565140 (599h7m1.485648175s) [noise=true , address=145.79.227.34:4907 , piece_id=KOIDZ5KYREDVLCDLCD7L47SHBXNZUU4CBHRYMHSLGTSL5H3EV7HQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614585379 (599h7m1.485668424s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614597231 (599h7m1.485680266s) [noise=true , address=45.91.116.13:4155 , piece_id=GLPX6WI4VAPF42HOZJFSNXQSYCMTCQA5RLWENSQISPXCEIYOQCTQ, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614857698 (599h7m1.485940743s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 30 14:36:01.614867877 (599h7m1.485950922s) [noise=true , address=77.90.31.10:4519 , piece_id=LYO6G5RUKMAERKY5GLQ2AQMR2K4VIUXVMLEQ6T22RCHG3RD2V57Q, seg_pos={0 0}, upload=525123]: Long tail cancellation due to optimal threshold
Dec 30 14:36:01.614894368 (599h7m1.485977403s) [seg_pos={0 0}, upload=525123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:36:01.614989599 (599h7m1.486072634s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:36:01.614994027 (599h7m1.486077063s) [upload=525123]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:36:01.905271368 (599h7m1.776354423s) [upload=525123]: Done uploading segments...
Dec 30 14:36:01.905339718 (599h7m1.776422763s) [upload=525123]: Upload finished. err: <nil>
Duration: 30.002707908s
core
uplink
us-select-1
|