Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [X....X.X....................................X...X.X.XX....X.] duration:
16.745366342s trace:
https://jaeger.dev.storj.io/trace/485874e9185c477e
FAIL (2025-12-29 13:51:26 UTC)
orbiter warning: context deadline exceeded
Duration: 16.745431977s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
43.151969443s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:51:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 43.151985733s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X...........X...X.] duration:
25.471910406s trace:
https://jaeger.dev.storj.io/trace/28971474a20c1c8
FAIL (2025-12-29 13:51:19 UTC)
orbiter warning: context deadline exceeded
Duration: 25.47192814s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X....X.X....................................X...X.X.XX....X] duration:
30.001497768s trace:
https://jaeger.dev.storj.io/trace/485874e9185c477e
FAIL (2025-12-29 13:51:10 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001513438s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [X..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.] duration:
10.079841947s trace:
https://jaeger.dev.storj.io/trace/3bb0b630d1e02238
FAIL (2025-12-29 13:50:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
c 29 13:49:43.629197261 (574h20m43.500280296s) [upload=503520]: Starting upload
Dec 29 13:49:43.629299666 (574h20m43.500382711s) [upload=503520]: Done starting upload
Dec 29 13:49:43.629439062 (574h20m43.500522107s) [upload=503520]: Uploading segments...
Dec 29 13:49:43.629554713 (574h20m43.500637748s) [upload=503520]: Got next segment. Inline: false
Dec 29 13:49:43.629615928 (574h20m43.500698974s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:49:43.629621690 (574h20m43.500704725s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:49:43.876874065 (574h20m43.7479571s) [seg_pos={0 0}, upload=503520]: Begin upload segment...
Dec 29 13:49:43.876919872 (574h20m43.748002907s) [seg_pos={0 0}, upload=503520]: Done begin upload segment.
Dec 29 13:49:43.876936854 (574h20m43.74801989s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.876953826 (574h20m43.748036861s) [noise=true , address=145.79.227.4:4185 , piece_id=YHAIOUUGZFYL47CER6FCCMQ4E74QUU76ETOFS47CYTLI2PTOI7BQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877348408 (574h20m43.748431443s) [noise=true , address=145.79.227.15:4875 , piece_id=DZM25OOSA3ZIGHZDSJSFX4J2EPJ3ARTFJJRORSIWZHQWPM2ACG6A, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877622770 (574h20m43.748705805s) [noise=true , address=45.91.116.12:4227 , piece_id=3WKMXGDFEIJ4UT7XTY2YQZVQ6LNUCY2HQR4Z5YM6CLXA2EFPT2XQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877933642 (574h20m43.749016687s) [noise=true , address=145.79.227.14:4839 , piece_id=7YNRGURMJN3F424ITYL3E4Y4VWGEU36H5TD3NW7SH7BIOBCVD64A, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878310600 (574h20m43.749393635s) [noise=true , address=145.79.227.3:4149 , piece_id=IFVKBRV7BE3EKCYQ35GNISDRLY5DPXG56MAJT2XBDYXECA5JKSQA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878580374 (574h20m43.749663419s) [noise=true , address=145.79.227.50:4373 , piece_id=PFIOYNFHNVRGVSLKAKEO3F65RXXHVWTUVRNOFGFV4SN2E5NCPV5Q, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878878691 (574h20m43.749961726s) [noise=true , address=77.90.31.21:4878 , piece_id=6WQKOJE6HQUUAXYXY6TPXNMI5D2R6YX7AJBIW5NG7T7JE2OEVBUQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.879372693 (574h20m43.750455728s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.880472206 (574h20m43.751555241s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.881768714 (574h20m43.752851749s) [noise=true , address=145.79.227.36:4519 , piece_id=2UO353XYGWCP5VON2ENTMZ5Q7YNVDQ5BYU4M3JZTF2GNYAOYEHMQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.883024116 (574h20m43.754107161s) [noise=true , address=145.79.227.52:4445 , piece_id=ANGU4FOCWWELZJUK4JONH2CVKLTGJ6KBZQJBIKKI62ZQ5R4AM3EQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.884225553 (574h20m43.755308588s) [noise=true , address=145.79.227.39:4013 , piece_id=3QE7PI2KVIW2TOAW32GSC4RDXVELFRACNACXJPXU5TSCSDZDMM3Q, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.886175596 (574h20m43.757258641s) [noise=true , address=145.79.227.5:4223 , piece_id=VQKMNOVGIXRFZ7JZTWVJ3ZJJRCMFHFDCAC4IMOUCOOBIQI2YV5FQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.886510655 (574h20m43.75759369s) [noise=true , address=145.79.227.37:4565 , piece_id=3AQWDCICYZKFXEZEJUWN7YQYRONWWCR6YIQSAT7RHKKGFLVTHPDA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.887557919 (574h20m43.758640954s) [noise=true , address=145.79.227.6:4257 , piece_id=5C3MCDRGPWWJOWT64H6P5PJOAUW6VDLYN37YJLDMW2ZQ45HLLHTQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:45.582915750 (574h20m45.453998795s) [upload=503520]: Next returned nil segment
Dec 29 13:49:45.582933984 (574h20m45.454017029s) [upload=503520]: Waiting for error group managing segments...
Dec 29 13:49:45.863598962 (574h20m45.734681997s) [noise=true , address=145.79.227.15:4875 , piece_id=DZM25OOSA3ZIGHZDSJSFX4J2EPJ3ARTFJJRORSIWZHQWPM2ACG6A, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.875938688 (574h20m45.747021723s) [noise=true , address=145.79.227.37:4565 , piece_id=3AQWDCICYZKFXEZEJUWN7YQYRONWWCR6YIQSAT7RHKKGFLVTHPDA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.876231846 (574h20m45.747314881s) [noise=true , address=145.79.227.3:4149 , piece_id=IFVKBRV7BE3EKCYQ35GNISDRLY5DPXG56MAJT2XBDYXECA5JKSQA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.877666438 (574h20m45.748749473s) [noise=true , address=145.79.227.50:4373 , piece_id=PFIOYNFHNVRGVSLKAKEO3F65RXXHVWTUVRNOFGFV4SN2E5NCPV5Q, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.878932188 (574h20m45.750015223s) [noise=true , address=145.79.227.36:4519 , piece_id=2UO353XYGWCP5VON2ENTMZ5Q7YNVDQ5BYU4M3JZTF2GNYAOYEHMQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.880266769 (574h20m45.751349804s) [noise=true , address=145.79.227.14:4839 , piece_id=7YNRGURMJN3F424ITYL3E4Y4VWGEU36H5TD3NW7SH7BIOBCVD64A, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.881874541 (574h20m45.752957576s) [noise=true , address=145.79.227.5:4223 , piece_id=VQKMNOVGIXRFZ7JZTWVJ3ZJJRCMFHFDCAC4IMOUCOOBIQI2YV5FQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.884282095 (574h20m45.75536514s) [noise=true , address=145.79.227.4:4185 , piece_id=YHAIOUUGZFYL47CER6FCCMQ4E74QUU76ETOFS47CYTLI2PTOI7BQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.897182379 (574h20m45.768265424s) [noise=true , address=145.79.227.39:4013 , piece_id=3QE7PI2KVIW2TOAW32GSC4RDXVELFRACNACXJPXU5TSCSDZDMM3Q, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.898975904 (574h20m45.770058949s) [noise=true , address=145.79.227.6:4257 , piece_id=5C3MCDRGPWWJOWT64H6P5PJOAUW6VDLYN37YJLDMW2ZQ45HLLHTQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.929185778 (574h20m45.800268813s) [noise=true , address=145.79.227.52:4445 , piece_id=ANGU4FOCWWELZJUK4JONH2CVKLTGJ6KBZQJBIKKI62ZQ5R4AM3EQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.090042884 (574h20m45.961125929s) [noise=true , address=77.90.31.21:4878 , piece_id=6WQKOJE6HQUUAXYXY6TPXNMI5D2R6YX7AJBIW5NG7T7JE2OEVBUQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.138711617 (574h20m46.009794662s) [noise=true , address=45.91.116.12:4227 , piece_id=3WKMXGDFEIJ4UT7XTY2YQZVQ6LNUCY2HQR4Z5YM6CLXA2EFPT2XQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.138736114 (574h20m46.009819159s) [seg_pos={0 0}, upload=503520]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:49:46.139002079 (574h20m46.010085125s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection
Dec 29 13:49:46.139010355 (574h20m46.0100934s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139064549 (574h20m46.010147584s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:46.139079186 (574h20m46.010162221s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139324593 (574h20m46.010407638s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:46.139333601 (574h20m46.010416636s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139357817 (574h20m46.010440852s) [seg_pos={0 0}, upload=503520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:49:46.139450694 (574h20m46.010533739s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:49:46.139456284 (574h20m46.010539319s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:49:46.433525335 (574h20m46.30460838s) [upload=503520]: Done uploading segments...
Dec 29 13:49:46.433548259 (574h20m46.304631294s) [upload=503520]: Upload finished. err: <nil>
Duration: 10.079874978s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX] duration:
5.00101885s trace:
https://jaeger.dev.storj.io/trace/4774ab38d2c0a3d3
FAIL (2025-12-29 13:50:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001041874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X...........X...X] duration:
1m0.000788626s trace:
https://jaeger.dev.storj.io/trace/28971474a20c1c8
FAIL (2025-12-29 13:50:53 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.0008124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
49.602803479s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:50:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 49.602819048s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXX] duration:
5.001381655s trace:
https://jaeger.dev.storj.io/trace/17769cf20a136062
FAIL (2025-12-29 13:50:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001407715s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.XXXXX] duration:
30.003317733s trace:
https://jaeger.dev.storj.io/trace/3bb0b630d1e02238
FAIL (2025-12-29 13:50:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
c 29 13:49:43.629197261 (574h20m43.500280296s) [upload=503520]: Starting upload
Dec 29 13:49:43.629299666 (574h20m43.500382711s) [upload=503520]: Done starting upload
Dec 29 13:49:43.629439062 (574h20m43.500522107s) [upload=503520]: Uploading segments...
Dec 29 13:49:43.629554713 (574h20m43.500637748s) [upload=503520]: Got next segment. Inline: false
Dec 29 13:49:43.629615928 (574h20m43.500698974s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:49:43.629621690 (574h20m43.500704725s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:49:43.876874065 (574h20m43.7479571s) [seg_pos={0 0}, upload=503520]: Begin upload segment...
Dec 29 13:49:43.876919872 (574h20m43.748002907s) [seg_pos={0 0}, upload=503520]: Done begin upload segment.
Dec 29 13:49:43.876936854 (574h20m43.74801989s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.876953826 (574h20m43.748036861s) [noise=true , address=145.79.227.4:4185 , piece_id=YHAIOUUGZFYL47CER6FCCMQ4E74QUU76ETOFS47CYTLI2PTOI7BQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877348408 (574h20m43.748431443s) [noise=true , address=145.79.227.15:4875 , piece_id=DZM25OOSA3ZIGHZDSJSFX4J2EPJ3ARTFJJRORSIWZHQWPM2ACG6A, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877622770 (574h20m43.748705805s) [noise=true , address=45.91.116.12:4227 , piece_id=3WKMXGDFEIJ4UT7XTY2YQZVQ6LNUCY2HQR4Z5YM6CLXA2EFPT2XQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.877933642 (574h20m43.749016687s) [noise=true , address=145.79.227.14:4839 , piece_id=7YNRGURMJN3F424ITYL3E4Y4VWGEU36H5TD3NW7SH7BIOBCVD64A, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878310600 (574h20m43.749393635s) [noise=true , address=145.79.227.3:4149 , piece_id=IFVKBRV7BE3EKCYQ35GNISDRLY5DPXG56MAJT2XBDYXECA5JKSQA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878580374 (574h20m43.749663419s) [noise=true , address=145.79.227.50:4373 , piece_id=PFIOYNFHNVRGVSLKAKEO3F65RXXHVWTUVRNOFGFV4SN2E5NCPV5Q, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.878878691 (574h20m43.749961726s) [noise=true , address=77.90.31.21:4878 , piece_id=6WQKOJE6HQUUAXYXY6TPXNMI5D2R6YX7AJBIW5NG7T7JE2OEVBUQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.879372693 (574h20m43.750455728s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.880472206 (574h20m43.751555241s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.881768714 (574h20m43.752851749s) [noise=true , address=145.79.227.36:4519 , piece_id=2UO353XYGWCP5VON2ENTMZ5Q7YNVDQ5BYU4M3JZTF2GNYAOYEHMQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.883024116 (574h20m43.754107161s) [noise=true , address=145.79.227.52:4445 , piece_id=ANGU4FOCWWELZJUK4JONH2CVKLTGJ6KBZQJBIKKI62ZQ5R4AM3EQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.884225553 (574h20m43.755308588s) [noise=true , address=145.79.227.39:4013 , piece_id=3QE7PI2KVIW2TOAW32GSC4RDXVELFRACNACXJPXU5TSCSDZDMM3Q, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.886175596 (574h20m43.757258641s) [noise=true , address=145.79.227.5:4223 , piece_id=VQKMNOVGIXRFZ7JZTWVJ3ZJJRCMFHFDCAC4IMOUCOOBIQI2YV5FQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.886510655 (574h20m43.75759369s) [noise=true , address=145.79.227.37:4565 , piece_id=3AQWDCICYZKFXEZEJUWN7YQYRONWWCR6YIQSAT7RHKKGFLVTHPDA, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:43.887557919 (574h20m43.758640954s) [noise=true , address=145.79.227.6:4257 , piece_id=5C3MCDRGPWWJOWT64H6P5PJOAUW6VDLYN37YJLDMW2ZQ45HLLHTQ, seg_pos={0 0}, upload=503520]: Uploading piece...
Dec 29 13:49:45.582915750 (574h20m45.453998795s) [upload=503520]: Next returned nil segment
Dec 29 13:49:45.582933984 (574h20m45.454017029s) [upload=503520]: Waiting for error group managing segments...
Dec 29 13:49:45.863598962 (574h20m45.734681997s) [noise=true , address=145.79.227.15:4875 , piece_id=DZM25OOSA3ZIGHZDSJSFX4J2EPJ3ARTFJJRORSIWZHQWPM2ACG6A, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.875938688 (574h20m45.747021723s) [noise=true , address=145.79.227.37:4565 , piece_id=3AQWDCICYZKFXEZEJUWN7YQYRONWWCR6YIQSAT7RHKKGFLVTHPDA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.876231846 (574h20m45.747314881s) [noise=true , address=145.79.227.3:4149 , piece_id=IFVKBRV7BE3EKCYQ35GNISDRLY5DPXG56MAJT2XBDYXECA5JKSQA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.877666438 (574h20m45.748749473s) [noise=true , address=145.79.227.50:4373 , piece_id=PFIOYNFHNVRGVSLKAKEO3F65RXXHVWTUVRNOFGFV4SN2E5NCPV5Q, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.878932188 (574h20m45.750015223s) [noise=true , address=145.79.227.36:4519 , piece_id=2UO353XYGWCP5VON2ENTMZ5Q7YNVDQ5BYU4M3JZTF2GNYAOYEHMQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.880266769 (574h20m45.751349804s) [noise=true , address=145.79.227.14:4839 , piece_id=7YNRGURMJN3F424ITYL3E4Y4VWGEU36H5TD3NW7SH7BIOBCVD64A, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.881874541 (574h20m45.752957576s) [noise=true , address=145.79.227.5:4223 , piece_id=VQKMNOVGIXRFZ7JZTWVJ3ZJJRCMFHFDCAC4IMOUCOOBIQI2YV5FQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.884282095 (574h20m45.75536514s) [noise=true , address=145.79.227.4:4185 , piece_id=YHAIOUUGZFYL47CER6FCCMQ4E74QUU76ETOFS47CYTLI2PTOI7BQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.897182379 (574h20m45.768265424s) [noise=true , address=145.79.227.39:4013 , piece_id=3QE7PI2KVIW2TOAW32GSC4RDXVELFRACNACXJPXU5TSCSDZDMM3Q, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.898975904 (574h20m45.770058949s) [noise=true , address=145.79.227.6:4257 , piece_id=5C3MCDRGPWWJOWT64H6P5PJOAUW6VDLYN37YJLDMW2ZQ45HLLHTQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:45.929185778 (574h20m45.800268813s) [noise=true , address=145.79.227.52:4445 , piece_id=ANGU4FOCWWELZJUK4JONH2CVKLTGJ6KBZQJBIKKI62ZQ5R4AM3EQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.090042884 (574h20m45.961125929s) [noise=true , address=77.90.31.21:4878 , piece_id=6WQKOJE6HQUUAXYXY6TPXNMI5D2R6YX7AJBIW5NG7T7JE2OEVBUQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.138711617 (574h20m46.009794662s) [noise=true , address=45.91.116.12:4227 , piece_id=3WKMXGDFEIJ4UT7XTY2YQZVQ6LNUCY2HQR4Z5YM6CLXA2EFPT2XQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: <nil>
Dec 29 13:49:46.138736114 (574h20m46.009819159s) [seg_pos={0 0}, upload=503520]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:49:46.139002079 (574h20m46.010085125s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57230->45.91.116.10:4084: use of closed network connection
Dec 29 13:49:46.139010355 (574h20m46.0100934s) [noise=true , address=45.91.116.10:4084 , piece_id=PZBKLQBC2SWFRVGMSAXTJFVAXYHZLHU3JBI4WND5BHFGS3PROKUA, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139064549 (574h20m46.010147584s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:46.139079186 (574h20m46.010162221s) [noise=true , address=77.90.31.11:4555 , piece_id=MCPOBHAAWOJ66YFFLWJYJHHBS63VINQ3A4BPNUNLVJA24PUPDCXQ, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139324593 (574h20m46.010407638s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:46.139333601 (574h20m46.010416636s) [noise=true , address=45.91.116.7:4011 , piece_id=KZOVOQOSOZFQMPFPG6VT5KVCRJPF7O7NTALZIXKPYT43IJOJN5SQ, seg_pos={0 0}, upload=503520]: Long tail cancellation due to optimal threshold
Dec 29 13:49:46.139357817 (574h20m46.010440852s) [seg_pos={0 0}, upload=503520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:49:46.139450694 (574h20m46.010533739s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:49:46.139456284 (574h20m46.010539319s) [upload=503520]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:49:46.433525335 (574h20m46.30460838s) [upload=503520]: Done uploading segments...
Dec 29 13:49:46.433548259 (574h20m46.304631294s) [upload=503520]: Upload finished. err: <nil>
Duration: 30.003341076s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
53.208395379s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:49:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 53.208409747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [X.X....X.X....................................X...X.X.XX....] duration:
18.98908875s trace:
https://jaeger.dev.storj.io/trace/21f7513ff9b7e8e9
FAIL (2025-12-29 13:49:47 UTC)
orbiter warning: context deadline exceeded
Duration: 18.989120149s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.XXXX] duration:
30.004242659s trace:
https://jaeger.dev.storj.io/trace/2d5c3d6a053dd235
FAIL (2025-12-29 13:49:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: G3WBPNMG6PJY4PQXCNEF62RD2RQ23SKN4AQR2SF44D7HD5KMLJYQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: J5AQT4UX3NF7CPXAB2QEA2LOAMA5NB343CID5M367OSUX6SX2LBQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NZWWE6Q6WGRAYC44VYSL5Y6CISRPP3OEOJO6Q7WCXSUZKH3OSBRA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VXSEOXFRPOGGVYCV3ESNM22URUUHX75FILHHWZTFZZJZAOGVPGWQ) context deadline exceeded
======LOGS======
13:49:11.124680852 (574h20m10.995763887s) [upload=503513]: Starting upload
Dec 29 13:49:11.124738772 (574h20m10.995821807s) [upload=503513]: Done starting upload
Dec 29 13:49:11.124853240 (574h20m10.995936275s) [upload=503513]: Uploading segments...
Dec 29 13:49:11.124963370 (574h20m10.996046415s) [upload=503513]: Got next segment. Inline: false
Dec 29 13:49:11.125019296 (574h20m10.996102331s) [upload=503513]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:49:11.125024456 (574h20m10.996107491s) [upload=503513]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:49:11.373301061 (574h20m11.244384107s) [seg_pos={0 0}, upload=503513]: Begin upload segment...
Dec 29 13:49:11.373358711 (574h20m11.244441746s) [seg_pos={0 0}, upload=503513]: Done begin upload segment.
Dec 29 13:49:11.373400770 (574h20m11.244483816s) [noise=true , address=145.79.227.11:4730 , piece_id=LRY5AP23EVJFMRA6NP3G5N5JNPBV3Y2IZUOXLNEPRDDERXJS2QZQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.373497665 (574h20m11.24458071s) [noise=true , address=45.91.116.10:4082 , piece_id=VXSEOXFRPOGGVYCV3ESNM22URUUHX75FILHHWZTFZZJZAOGVPGWQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.374168764 (574h20m11.245251809s) [noise=true , address=145.79.227.39:4014 , piece_id=HOVIYAXTX3LSA4KJJ2CMPUMP4X7MZNU6I7WG46KSBYQUOHNUAYJQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.374612548 (574h20m11.245695583s) [noise=true , address=145.79.227.17:4970 , piece_id=3WNJOQEKPVGRLG7SW4WDZ2LJT46Q4KSKEJVC7W3V56KO6KL35NPQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.375813976 (574h20m11.246897021s) [noise=true , address=145.79.227.24:4582 , piece_id=7SE6Y4PY6F6NGQ6J5TL4PNLJQDRVBYE5H7YARGYV7XHP4WWOBC3Q, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.376295373 (574h20m11.247378408s) [noise=true , address=77.90.31.13:4629 , piece_id=RVWENG4YE3QHVDDRSEN6U6TEAZALIMZFQT53NSJNDVZL3CJQ7AVA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.376719931 (574h20m11.247802976s) [noise=true , address=45.91.116.7:4009 , piece_id=GTT3A2VGDD47MGGSF2HA77WJP2A3EOQWBQQRPXQMXHX5ST3KZOKA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.377252896 (574h20m11.248335931s) [noise=true , address=145.79.227.51:4410 , piece_id=6ZYOBIXKPPO7LUCCO3VL4L5FCEUX7OEOWCG6FWGEJEL7O7QKBSXQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.377701490 (574h20m11.248784535s) [noise=true , address=45.91.116.13:4156 , piece_id=NNV2PZWR7GFTGDWFKO5SKHXDR4QPUAQ5MLZ3T5VLBYM7TWLKD5HA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.378708217 (574h20m11.249791252s) [noise=true , address=145.79.227.45:4230 , piece_id=JWNMO4PVGVMNEAHAXGSLR5VRZ4QSIAP3PL2SJHIAE4JTY6OZV2CA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.379253094 (574h20m11.250336139s) [noise=true , address=145.79.227.26:4655 , piece_id=GN43TTWNYBCRGVQPXP6R65PC4Y545OX5FP4U7YGT2Q5PHNITRCSA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.379745332 (574h20m11.250828367s) [noise=true , address=77.90.31.15:4697 , piece_id=G3WBPNMG6PJY4PQXCNEF62RD2RQ23SKN4AQR2SF44D7HD5KMLJYQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.379968347 (574h20m11.251051392s) [noise=true , address=145.79.227.48:4303 , piece_id=UTLZIS6VS2RVLVOWDBCJCPDYOC6L4O2QRVYENJF6RDTAEFFPYZGQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.380764743 (574h20m11.251847778s) [noise=true , address=145.79.227.3:4149 , piece_id=N53VBQ7D2GJ24Z4DKDP2DM3MB3O2NW5OAQOGGWJGSGEE5EQZVKTQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.381414239 (574h20m11.252497284s) [noise=true , address=45.91.116.12:4226 , piece_id=J5AQT4UX3NF7CPXAB2QEA2LOAMA5NB343CID5M367OSUX6SX2LBQ, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:11.388223466 (574h20m11.259306501s) [noise=true , address=45.91.116.5:4299 , piece_id=NZWWE6Q6WGRAYC44VYSL5Y6CISRPP3OEOJO6Q7WCXSUZKH3OSBRA, seg_pos={0 0}, upload=503513]: Uploading piece...
Dec 29 13:49:12.975603321 (574h20m12.846686356s) [upload=503513]: Next returned nil segment
Dec 29 13:49:12.975618820 (574h20m12.846701855s) [upload=503513]: Waiting for error group managing segments...
Dec 29 13:49:13.357051634 (574h20m13.228134669s) [noise=true , address=145.79.227.11:4730 , piece_id=LRY5AP23EVJFMRA6NP3G5N5JNPBV3Y2IZUOXLNEPRDDERXJS2QZQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.358504821 (574h20m13.229587856s) [noise=true , address=145.79.227.17:4970 , piece_id=3WNJOQEKPVGRLG7SW4WDZ2LJT46Q4KSKEJVC7W3V56KO6KL35NPQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.362625999 (574h20m13.233709034s) [noise=true , address=145.79.227.26:4655 , piece_id=GN43TTWNYBCRGVQPXP6R65PC4Y545OX5FP4U7YGT2Q5PHNITRCSA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.372258450 (574h20m13.243341495s) [noise=true , address=145.79.227.45:4230 , piece_id=JWNMO4PVGVMNEAHAXGSLR5VRZ4QSIAP3PL2SJHIAE4JTY6OZV2CA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.385590086 (574h20m13.256673121s) [noise=true , address=145.79.227.3:4149 , piece_id=N53VBQ7D2GJ24Z4DKDP2DM3MB3O2NW5OAQOGGWJGSGEE5EQZVKTQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.388106687 (574h20m13.259189733s) [noise=true , address=145.79.227.24:4582 , piece_id=7SE6Y4PY6F6NGQ6J5TL4PNLJQDRVBYE5H7YARGYV7XHP4WWOBC3Q, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.392299351 (574h20m13.263382396s) [noise=true , address=145.79.227.51:4410 , piece_id=6ZYOBIXKPPO7LUCCO3VL4L5FCEUX7OEOWCG6FWGEJEL7O7QKBSXQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.392641332 (574h20m13.263724367s) [noise=true , address=145.79.227.39:4014 , piece_id=HOVIYAXTX3LSA4KJJ2CMPUMP4X7MZNU6I7WG46KSBYQUOHNUAYJQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.393304144 (574h20m13.264387179s) [noise=true , address=145.79.227.48:4303 , piece_id=UTLZIS6VS2RVLVOWDBCJCPDYOC6L4O2QRVYENJF6RDTAEFFPYZGQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.660119370 (574h20m13.531202405s) [noise=true , address=77.90.31.15:4697 , piece_id=G3WBPNMG6PJY4PQXCNEF62RD2RQ23SKN4AQR2SF44D7HD5KMLJYQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.734963213 (574h20m13.606046258s) [noise=true , address=45.91.116.5:4299 , piece_id=NZWWE6Q6WGRAYC44VYSL5Y6CISRPP3OEOJO6Q7WCXSUZKH3OSBRA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:13.807380777 (574h20m13.678463812s) [noise=true , address=45.91.116.10:4082 , piece_id=VXSEOXFRPOGGVYCV3ESNM22URUUHX75FILHHWZTFZZJZAOGVPGWQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:14.029286188 (574h20m13.900369233s) [noise=true , address=45.91.116.12:4226 , piece_id=J5AQT4UX3NF7CPXAB2QEA2LOAMA5NB343CID5M367OSUX6SX2LBQ, seg_pos={0 0}, upload=503513]: Done uploading piece. err: <nil>
Dec 29 13:49:14.029312908 (574h20m13.900395953s) [seg_pos={0 0}, upload=503513]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:49:14.029746895 (574h20m13.90082994s) [noise=true , address=77.90.31.13:4629 , piece_id=RVWENG4YE3QHVDDRSEN6U6TEAZALIMZFQT53NSJNDVZL3CJQ7AVA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:14.029759829 (574h20m13.900842864s) [noise=true , address=77.90.31.13:4629 , piece_id=RVWENG4YE3QHVDDRSEN6U6TEAZALIMZFQT53NSJNDVZL3CJQ7AVA, seg_pos={0 0}, upload=503513]: Long tail cancellation due to optimal threshold
Dec 29 13:49:14.029783074 (574h20m13.900866119s) [noise=true , address=45.91.116.13:4156 , piece_id=NNV2PZWR7GFTGDWFKO5SKHXDR4QPUAQ5MLZ3T5VLBYM7TWLKD5HA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38008->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38008->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38008->45.91.116.13:4156: use of closed network connection
Dec 29 13:49:14.029847607 (574h20m13.900930642s) [noise=true , address=45.91.116.13:4156 , piece_id=NNV2PZWR7GFTGDWFKO5SKHXDR4QPUAQ5MLZ3T5VLBYM7TWLKD5HA, seg_pos={0 0}, upload=503513]: Long tail cancellation due to optimal threshold
Dec 29 13:49:14.030244442 (574h20m13.901327517s) [noise=true , address=45.91.116.7:4009 , piece_id=GTT3A2VGDD47MGGSF2HA77WJP2A3EOQWBQQRPXQMXHX5ST3KZOKA, seg_pos={0 0}, upload=503513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 13:49:14.030256925 (574h20m13.901339961s) [noise=true , address=45.91.116.7:4009 , piece_id=GTT3A2VGDD47MGGSF2HA77WJP2A3EOQWBQQRPXQMXHX5ST3KZOKA, seg_pos={0 0}, upload=503513]: Long tail cancellation due to optimal threshold
Dec 29 13:49:14.030289858 (574h20m13.901372903s) [seg_pos={0 0}, upload=503513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:49:14.030389749 (574h20m13.901472784s) [upload=503513]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:49:14.030420237 (574h20m13.901503282s) [upload=503513]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:49:14.328100744 (574h20m14.19918379s) [upload=503513]: Done uploading segments...
Dec 29 13:49:14.328127385 (574h20m14.19921043s) [upload=503513]: Upload finished. err: <nil>
Duration: 30.004264341s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................X...........X...] duration:
20.816213602s trace:
https://jaeger.dev.storj.io/trace/7abcd5a2f6a75b6d
FAIL (2025-12-29 13:49:33 UTC)
orbiter warning: context deadline exceeded
Duration: 20.816242607s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXX] duration:
5.001158041s trace:
https://jaeger.dev.storj.io/trace/1a008595d1ba600f
FAIL (2025-12-29 13:49:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001182106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X...........X..] duration:
20.583810103s trace:
https://jaeger.dev.storj.io/trace/7abcd5a2f6a75b6d
FAIL (2025-12-29 13:49:00 UTC)
orbiter warning: context deadline exceeded
Duration: 20.583833066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................XX.........................................] duration:
49.078736056s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:48:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 49.078758288s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.XXX] duration:
30.002360141s trace:
https://jaeger.dev.storj.io/trace/ca1860cae26dc08
FAIL (2025-12-29 13:48:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: BDTM2IADTZFYTON5UTQ7U3CVEEWQZFSZVB4A2K7GNX2FZL3IMVKQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Y2A7AADMFIVVZ5G7WBRSS54PM7FG5J3A35CV2CHXKQQ4IDKKY6LQ) context deadline exceeded
======LOGS======
Dec 29 13:48:10.090243279 (574h19m9.961326314s) [upload=503499]: Starting upload
Dec 29 13:48:10.090401370 (574h19m9.961484416s) [upload=503499]: Done starting upload
Dec 29 13:48:10.090548841 (574h19m9.961631896s) [upload=503499]: Uploading segments...
Dec 29 13:48:10.090719937 (574h19m9.961802982s) [upload=503499]: Got next segment. Inline: false
Dec 29 13:48:10.090808847 (574h19m9.961891892s) [upload=503499]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:48:10.090816110 (574h19m9.961899145s) [upload=503499]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:48:10.362835103 (574h19m10.233918158s) [seg_pos={0 0}, upload=503499]: Begin upload segment...
Dec 29 13:48:10.362970822 (574h19m10.234053857s) [seg_pos={0 0}, upload=503499]: Done begin upload segment.
Dec 29 13:48:10.363011911 (574h19m10.234094956s) [noise=true , address=145.79.227.51:4410 , piece_id=HRWLFLKDIRVZVJTSWLBQKCP5OO3N3LAUXBGCGD764H55XJAKGBGA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.364056800 (574h19m10.235139846s) [noise=true , address=77.90.31.11:4554 , piece_id=BDTM2IADTZFYTON5UTQ7U3CVEEWQZFSZVB4A2K7GNX2FZL3IMVKQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.364995467 (574h19m10.236078502s) [noise=true , address=145.79.227.47:4267 , piece_id=G5IFASIEUOWJCMGR2VHTWBQCRQPI2GOD7GYS7HQKR2CA3IUMQAXA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.365429834 (574h19m10.23651288s) [noise=true , address=145.79.227.29:4943 , piece_id=DIOB7AILLQ3CIISCAW4JUC7ABFGOLJTHAR2YHFEHMVQN5CEKLHSA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.365962799 (574h19m10.237045844s) [noise=true , address=77.90.31.21:4880 , piece_id=Y2A7AADMFIVVZ5G7WBRSS54PM7FG5J3A35CV2CHXKQQ4IDKKY6LQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.366427044 (574h19m10.237510079s) [noise=true , address=145.79.227.24:4582 , piece_id=AACA43NDRQ64XRNKUJAOWHHIMAN6BQFKBUP327EYMUW5GXUC4LIQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.366759216 (574h19m10.237842251s) [noise=true , address=145.79.227.38:4977 , piece_id=ZJKUULXIMKX64K2H2VMZVBMAFN3UG3YDGYMHEMDONXM5B7AJWM5Q, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.367484006 (574h19m10.238567041s) [noise=true , address=145.79.227.4:4186 , piece_id=3D57GR2XV23ERG4G4LGG3X4RQW47QQEMDEXS6NWZFGR4SYDYEJFQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.368102644 (574h19m10.239185679s) [noise=true , address=145.79.227.17:4971 , piece_id=OYJ7ZJLLAPDHAHQYHQNXQG76C2KTOT7SSBGFR7MIXTHXB2UF7LPQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.368928386 (574h19m10.240011421s) [noise=true , address=145.79.227.48:4301 , piece_id=W23YM6PMAP3FPS2FZ3UHQ2PJNQWZA4KBHBTJOMTLTOSWS2PIVMWA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.369501678 (574h19m10.240584713s) [noise=true , address=145.79.227.36:4519 , piece_id=B4DS2DDSZH7CMQ3FPMNQKKPTLVWS3HZSXBKPMWLHRWF4D4DLKVIQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.369954580 (574h19m10.241037625s) [noise=true , address=145.79.227.41:4086 , piece_id=S4LCNLNNSD4C2XS633XU4JFVOUBEXZAVACBANR5BUQ3ZPBYSI5SQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.370919057 (574h19m10.242002102s) [noise=true , address=45.91.116.9:4263 , piece_id=PUUZ3KACIFLFUKDVCTD5X4ZFCRWIDTCNFTLQISATR5XX47LVOJRQ, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.371027083 (574h19m10.242110118s) [noise=true , address=145.79.227.19:4078 , piece_id=XAHGWCMSZJJXF2TN6D6GOHWKS3PO7RYVTHF2FBDB4SIWUJ56FSTA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.371531242 (574h19m10.242614277s) [noise=true , address=145.79.227.46:4602 , piece_id=GU2DSH2YESRNZDTXYOBOUX7GZNEQGZRQIV4M2LNJBDEORG6MKIHA, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:10.372498194 (574h19m10.243581239s) [noise=true , address=145.79.227.6:4258 , piece_id=MPLJT7VATK6RKYWJJ2DTVVVRCRQUUOAWWFHAIGH64LANBCXJIG5A, seg_pos={0 0}, upload=503499]: Uploading piece...
Dec 29 13:48:12.060818394 (574h19m11.931901429s) [upload=503499]: Next returned nil segment
Dec 29 13:48:12.060839444 (574h19m11.931922489s) [upload=503499]: Waiting for error group managing segments...
Dec 29 13:48:12.342007048 (574h19m12.213090083s) [noise=true , address=145.79.227.6:4258 , piece_id=MPLJT7VATK6RKYWJJ2DTVVVRCRQUUOAWWFHAIGH64LANBCXJIG5A, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.348703018 (574h19m12.219786063s) [noise=true , address=145.79.227.47:4267 , piece_id=G5IFASIEUOWJCMGR2VHTWBQCRQPI2GOD7GYS7HQKR2CA3IUMQAXA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.350253681 (574h19m12.221336726s) [noise=true , address=145.79.227.17:4971 , piece_id=OYJ7ZJLLAPDHAHQYHQNXQG76C2KTOT7SSBGFR7MIXTHXB2UF7LPQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.354221677 (574h19m12.225304722s) [noise=true , address=145.79.227.51:4410 , piece_id=HRWLFLKDIRVZVJTSWLBQKCP5OO3N3LAUXBGCGD764H55XJAKGBGA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.359079617 (574h19m12.230162652s) [noise=true , address=145.79.227.4:4186 , piece_id=3D57GR2XV23ERG4G4LGG3X4RQW47QQEMDEXS6NWZFGR4SYDYEJFQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.360357951 (574h19m12.231440996s) [noise=true , address=145.79.227.48:4301 , piece_id=W23YM6PMAP3FPS2FZ3UHQ2PJNQWZA4KBHBTJOMTLTOSWS2PIVMWA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.361041123 (574h19m12.232124168s) [noise=true , address=145.79.227.24:4582 , piece_id=AACA43NDRQ64XRNKUJAOWHHIMAN6BQFKBUP327EYMUW5GXUC4LIQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.376502482 (574h19m12.247585528s) [noise=true , address=145.79.227.41:4086 , piece_id=S4LCNLNNSD4C2XS633XU4JFVOUBEXZAVACBANR5BUQ3ZPBYSI5SQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.377191084 (574h19m12.248274129s) [noise=true , address=145.79.227.46:4602 , piece_id=GU2DSH2YESRNZDTXYOBOUX7GZNEQGZRQIV4M2LNJBDEORG6MKIHA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.380131843 (574h19m12.251214888s) [noise=true , address=145.79.227.19:4078 , piece_id=XAHGWCMSZJJXF2TN6D6GOHWKS3PO7RYVTHF2FBDB4SIWUJ56FSTA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.385157032 (574h19m12.256240067s) [noise=true , address=145.79.227.36:4519 , piece_id=B4DS2DDSZH7CMQ3FPMNQKKPTLVWS3HZSXBKPMWLHRWF4D4DLKVIQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:12.651290009 (574h19m12.522373054s) [noise=true , address=77.90.31.11:4554 , piece_id=BDTM2IADTZFYTON5UTQ7U3CVEEWQZFSZVB4A2K7GNX2FZL3IMVKQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:13.550301818 (574h19m13.421384853s) [noise=true , address=77.90.31.21:4880 , piece_id=Y2A7AADMFIVVZ5G7WBRSS54PM7FG5J3A35CV2CHXKQQ4IDKKY6LQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: <nil>
Dec 29 13:48:13.550326345 (574h19m13.42140938s) [seg_pos={0 0}, upload=503499]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:48:13.550578244 (574h19m13.421661279s) [noise=true , address=145.79.227.29:4943 , piece_id=DIOB7AILLQ3CIISCAW4JUC7ABFGOLJTHAR2YHFEHMVQN5CEKLHSA, seg_pos={0 0}, upload=503499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 29 13:48:13.550585449 (574h19m13.421668484s) [noise=true , address=145.79.227.29:4943 , piece_id=DIOB7AILLQ3CIISCAW4JUC7ABFGOLJTHAR2YHFEHMVQN5CEKLHSA, seg_pos={0 0}, upload=503499]: Long tail cancellation due to optimal threshold
Dec 29 13:48:13.550602310 (574h19m13.421685355s) [noise=true , address=45.91.116.9:4263 , piece_id=PUUZ3KACIFLFUKDVCTD5X4ZFCRWIDTCNFTLQISATR5XX47LVOJRQ, seg_pos={0 0}, upload=503499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:58546->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:58546->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58546->45.91.116.9:4263: use of closed network connection
Dec 29 13:48:13.550618832 (574h19m13.421701867s) [noise=true , address=45.91.116.9:4263 , piece_id=PUUZ3KACIFLFUKDVCTD5X4ZFCRWIDTCNFTLQISATR5XX47LVOJRQ, seg_pos={0 0}, upload=503499]: Long tail cancellation due to optimal threshold
Dec 29 13:48:13.550898004 (574h19m13.421981039s) [noise=true , address=145.79.227.38:4977 , piece_id=ZJKUULXIMKX64K2H2VMZVBMAFN3UG3YDGYMHEMDONXM5B7AJWM5Q, seg_pos={0 0}, upload=503499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF
Dec 29 13:48:13.550906379 (574h19m13.421989415s) [noise=true , address=145.79.227.38:4977 , piece_id=ZJKUULXIMKX64K2H2VMZVBMAFN3UG3YDGYMHEMDONXM5B7AJWM5Q, seg_pos={0 0}, upload=503499]: Long tail cancellation due to optimal threshold
Dec 29 13:48:13.550925225 (574h19m13.42200827s) [seg_pos={0 0}, upload=503499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:48:13.551045875 (574h19m13.42212892s) [upload=503499]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:48:13.551054942 (574h19m13.422137987s) [upload=503499]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:48:13.849153515 (574h19m13.72023655s) [upload=503499]: Done uploading segments...
Dec 29 13:48:13.849178773 (574h19m13.720261818s) [upload=503499]: Upload finished. err: <nil>
Duration: 30.002373917s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X.X....X.X....................................X...X.X.XX...] duration:
16.17589159s trace:
https://jaeger.dev.storj.io/trace/21f7513ff9b7e8e9
FAIL (2025-12-29 13:48:33 UTC)
orbiter warning: context deadline exceeded
Duration: 16.175910966s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX] duration:
5.001198966s trace:
https://jaeger.dev.storj.io/trace/45cf66113a010b32
FAIL (2025-12-29 13:48:33 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001220375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X.X....X.X....................................X...X.X.XX..] duration:
19.839274855s trace:
https://jaeger.dev.storj.io/trace/21f7513ff9b7e8e9
FAIL (2025-12-29 13:48:10 UTC)
orbiter warning: context deadline exceeded
Duration: 19.839301267s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X...........X.] duration:
13.731619038s trace:
https://jaeger.dev.storj.io/trace/7abcd5a2f6a75b6d
FAIL (2025-12-29 13:48:09 UTC)
orbiter warning: context deadline exceeded
Duration: 13.731635088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................XX........................................] duration:
58.134591388s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:48:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 58.134620273s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X...........X] duration:
1m0.001448277s trace:
https://jaeger.dev.storj.io/trace/7abcd5a2f6a75b6d
FAIL (2025-12-29 13:47:55 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001460499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.XX] duration:
30.004100953s trace:
https://jaeger.dev.storj.io/trace/4e2e9b73e419ff30
FAIL (2025-12-29 13:47:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: I3HHY4GKMZQRQZVKIAW5AAFOYEG4C7XKSCQZKXRRTIHWYTM4UIHA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: YVQHXFQAMC62LM27LGHFDJDFQAAIVMPLGYC2R3VHED7FDZUTBVYQ) context deadline exceeded
======LOGS======
ec 29 13:47:25.162432349 (574h18m25.033515385s) [upload=503487]: Starting upload
Dec 29 13:47:25.162494427 (574h18m25.033577462s) [upload=503487]: Done starting upload
Dec 29 13:47:25.162544573 (574h18m25.033627608s) [upload=503487]: Uploading segments...
Dec 29 13:47:25.162704787 (574h18m25.033787822s) [upload=503487]: Got next segment. Inline: false
Dec 29 13:47:25.162750585 (574h18m25.03383362s) [upload=503487]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:47:25.162754873 (574h18m25.033837908s) [upload=503487]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:47:25.409328987 (574h18m25.280412022s) [seg_pos={0 0}, upload=503487]: Begin upload segment...
Dec 29 13:47:25.409386896 (574h18m25.280469931s) [seg_pos={0 0}, upload=503487]: Done begin upload segment.
Dec 29 13:47:25.409420150 (574h18m25.280503185s) [noise=true , address=145.79.227.16:4912 , piece_id=AJURVOW6F7GEEYXXKXWGBZO7FSS5BLIRAC4Q2OD2FHHCAPVHN3EQ, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.409510252 (574h18m25.280593287s) [noise=true , address=145.79.227.48:4303 , piece_id=3ACD6JILTHWI27JLDJNEECUZXBXEWKHRYXA7BFDKH5OAH773GBOA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.410053956 (574h18m25.281136991s) [noise=true , address=45.91.116.9:4262 , piece_id=7BO4CFPL7EEAEXYBYR5HNEZ6FWRX5D7RMR3Y2P2JLG4LI2ZGX55Q, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.410549571 (574h18m25.281632606s) [noise=true , address=145.79.227.39:4013 , piece_id=YJVQYA7IGW33G22WPN5DMUTO2OL3H3VC3YATDNQKHW7ELAE4MWLA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.410915507 (574h18m25.281998542s) [noise=true , address=145.79.227.7:4294 , piece_id=IAYZJKEN43BVDOOOPIANRUX7TKJQSCZIKNDYXCVVXX54FAJVLD5Q, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.411254102 (574h18m25.282337137s) [noise=true , address=77.90.31.19:4841 , piece_id=YVQHXFQAMC62LM27LGHFDJDFQAAIVMPLGYC2R3VHED7FDZUTBVYQ, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.411577147 (574h18m25.282660192s) [noise=true , address=77.90.31.14:4664 , piece_id=3AHLQJB37BHPNK3H3Y5G3YL63RZ2MBM45O47DPNWPXULL5FUAZTQ, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.413079127 (574h18m25.284162162s) [noise=true , address=145.79.227.50:4375 , piece_id=MRF7DVDKMHQPCRFBW2UVS7IK5AKCRAW5R5NZXKSXZ5XESIMUTTNA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.414585877 (574h18m25.285668922s) [noise=true , address=145.79.227.51:4410 , piece_id=P7NVGSGK25JUIFMAVTGQ6RAV4TKIOPX5G7BW4IN6XY3TINUZHIVA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.415933703 (574h18m25.287016738s) [noise=true , address=145.79.227.19:4079 , piece_id=G4FFPFAOEFIB4BY35BOM7TSXWBZCF7JQAGHYNCVA3S53HL6NV7WQ, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.419113238 (574h18m25.290196273s) [noise=true , address=145.79.227.49:4338 , piece_id=WTHA2Z6BEL4BSX34ZPMOJDPSFBW3INRM4BGP4RQUQGROLVZTKTBA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.420668098 (574h18m25.291751143s) [noise=true , address=45.91.116.13:4153 , piece_id=24SDFPW4A63JP7CU23RZV7HUDBBLW2QH7ADRCCR3M454IC6AHSOA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.423749305 (574h18m25.29483234s) [noise=true , address=45.91.116.7:4011 , piece_id=6EIWGMV3OEE5VSMCO4PY4KLGPS6GTHPQI3QAKEIB7WF2T5MVRGHA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.426623528 (574h18m25.297706563s) [noise=true , address=145.79.227.47:4266 , piece_id=DLV453CSNNWBSZOYUJGIKZOXG65RDTWSZKEWM4U3QNWUWCCENQWA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.427649302 (574h18m25.298732337s) [noise=true , address=145.79.227.22:4511 , piece_id=AY3FR44I25IJFMHQE5BRXO7DGRDI2RV67P7NTDVZE5NY5R4FKXNQ, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:25.429081208 (574h18m25.300164273s) [noise=true , address=45.91.116.6:4975 , piece_id=I3HHY4GKMZQRQZVKIAW5AAFOYEG4C7XKSCQZKXRRTIHWYTM4UIHA, seg_pos={0 0}, upload=503487]: Uploading piece...
Dec 29 13:47:27.121996106 (574h18m26.993079151s) [upload=503487]: Next returned nil segment
Dec 29 13:47:27.122011445 (574h18m26.99309448s) [upload=503487]: Waiting for error group managing segments...
Dec 29 13:47:27.400858411 (574h18m27.271941447s) [noise=true , address=145.79.227.48:4303 , piece_id=3ACD6JILTHWI27JLDJNEECUZXBXEWKHRYXA7BFDKH5OAH773GBOA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.408555677 (574h18m27.279638712s) [noise=true , address=145.79.227.19:4079 , piece_id=G4FFPFAOEFIB4BY35BOM7TSXWBZCF7JQAGHYNCVA3S53HL6NV7WQ, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.410755757 (574h18m27.281838792s) [noise=true , address=145.79.227.7:4294 , piece_id=IAYZJKEN43BVDOOOPIANRUX7TKJQSCZIKNDYXCVVXX54FAJVLD5Q, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.411245470 (574h18m27.282328505s) [noise=true , address=145.79.227.39:4013 , piece_id=YJVQYA7IGW33G22WPN5DMUTO2OL3H3VC3YATDNQKHW7ELAE4MWLA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.413840400 (574h18m27.284923436s) [noise=true , address=145.79.227.22:4511 , piece_id=AY3FR44I25IJFMHQE5BRXO7DGRDI2RV67P7NTDVZE5NY5R4FKXNQ, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.414220054 (574h18m27.285303099s) [noise=true , address=145.79.227.51:4410 , piece_id=P7NVGSGK25JUIFMAVTGQ6RAV4TKIOPX5G7BW4IN6XY3TINUZHIVA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.415312654 (574h18m27.286395699s) [noise=true , address=145.79.227.50:4375 , piece_id=MRF7DVDKMHQPCRFBW2UVS7IK5AKCRAW5R5NZXKSXZ5XESIMUTTNA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.422542861 (574h18m27.293625907s) [noise=true , address=145.79.227.16:4912 , piece_id=AJURVOW6F7GEEYXXKXWGBZO7FSS5BLIRAC4Q2OD2FHHCAPVHN3EQ, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.427895132 (574h18m27.298978167s) [noise=true , address=145.79.227.47:4266 , piece_id=DLV453CSNNWBSZOYUJGIKZOXG65RDTWSZKEWM4U3QNWUWCCENQWA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.605734145 (574h18m27.47681718s) [noise=true , address=77.90.31.19:4841 , piece_id=YVQHXFQAMC62LM27LGHFDJDFQAAIVMPLGYC2R3VHED7FDZUTBVYQ, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.755013595 (574h18m27.62609665s) [noise=true , address=77.90.31.14:4664 , piece_id=3AHLQJB37BHPNK3H3Y5G3YL63RZ2MBM45O47DPNWPXULL5FUAZTQ, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.780263745 (574h18m27.65134679s) [noise=true , address=45.91.116.6:4975 , piece_id=I3HHY4GKMZQRQZVKIAW5AAFOYEG4C7XKSCQZKXRRTIHWYTM4UIHA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.995365831 (574h18m27.866448866s) [noise=true , address=45.91.116.9:4262 , piece_id=7BO4CFPL7EEAEXYBYR5HNEZ6FWRX5D7RMR3Y2P2JLG4LI2ZGX55Q, seg_pos={0 0}, upload=503487]: Done uploading piece. err: <nil>
Dec 29 13:47:27.995386640 (574h18m27.866469675s) [seg_pos={0 0}, upload=503487]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:47:27.995541275 (574h18m27.86662431s) [noise=true , address=45.91.116.7:4011 , piece_id=6EIWGMV3OEE5VSMCO4PY4KLGPS6GTHPQI3QAKEIB7WF2T5MVRGHA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 29 13:47:27.995547467 (574h18m27.866630522s) [noise=true , address=45.91.116.7:4011 , piece_id=6EIWGMV3OEE5VSMCO4PY4KLGPS6GTHPQI3QAKEIB7WF2T5MVRGHA, seg_pos={0 0}, upload=503487]: Long tail cancellation due to optimal threshold
Dec 29 13:47:27.995790169 (574h18m27.866873214s) [noise=true , address=145.79.227.49:4338 , piece_id=WTHA2Z6BEL4BSX34ZPMOJDPSFBW3INRM4BGP4RQUQGROLVZTKTBA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 29 13:47:27.995798745 (574h18m27.86688184s) [noise=true , address=145.79.227.49:4338 , piece_id=WTHA2Z6BEL4BSX34ZPMOJDPSFBW3INRM4BGP4RQUQGROLVZTKTBA, seg_pos={0 0}, upload=503487]: Long tail cancellation due to optimal threshold
Dec 29 13:47:27.995951185 (574h18m27.86703422s) [noise=true , address=45.91.116.13:4153 , piece_id=24SDFPW4A63JP7CU23RZV7HUDBBLW2QH7ADRCCR3M454IC6AHSOA, seg_pos={0 0}, upload=503487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:58762->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:58762->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58762->45.91.116.13:4153: use of closed network connection
Dec 29 13:47:27.995957948 (574h18m27.867040994s) [noise=true , address=45.91.116.13:4153 , piece_id=24SDFPW4A63JP7CU23RZV7HUDBBLW2QH7ADRCCR3M454IC6AHSOA, seg_pos={0 0}, upload=503487]: Long tail cancellation due to optimal threshold
Dec 29 13:47:27.995978778 (574h18m27.867061823s) [seg_pos={0 0}, upload=503487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:47:27.996062798 (574h18m27.867145863s) [upload=503487]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:47:27.996068729 (574h18m27.867151774s) [upload=503487]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:47:28.286804513 (574h18m28.157887559s) [upload=503487]: Done uploading segments...
Dec 29 13:47:28.286829009 (574h18m28.157912044s) [upload=503487]: Upload finished. err: <nil>
Duration: 30.004124838s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXX] duration:
5.001094559s trace:
https://jaeger.dev.storj.io/trace/367c4a4822b1faec
FAIL (2025-12-29 13:47:50 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001116842s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.X] duration:
30.004853449s trace:
https://jaeger.dev.storj.io/trace/458c7d3727a9348d
FAIL (2025-12-29 13:47:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:46:42.013339254 (574h17m41.884422299s) [upload=503477]: Starting upload
Dec 29 13:46:42.013403206 (574h17m41.884486271s) [upload=503477]: Done starting upload
Dec 29 13:46:42.013535949 (574h17m41.884618994s) [upload=503477]: Uploading segments...
Dec 29 13:46:42.013624237 (574h17m41.884707283s) [upload=503477]: Got next segment. Inline: false
Dec 29 13:46:42.013749856 (574h17m41.884832901s) [upload=503477]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:46:42.013754525 (574h17m41.88483756s) [upload=503477]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:46:42.266560203 (574h17m42.137643258s) [seg_pos={0 0}, upload=503477]: Begin upload segment...
Dec 29 13:46:42.266625017 (574h17m42.137708062s) [seg_pos={0 0}, upload=503477]: Done begin upload segment.
Dec 29 13:46:42.266678369 (574h17m42.137761404s) [noise=true , address=145.79.227.41:4086 , piece_id=T5OLI7E4ZPBB63EHXRYA76DRHOHZYVOYH37VCMBCJEUI26JZGANQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.266684080 (574h17m42.137767115s) [noise=true , address=145.79.227.26:4655 , piece_id=6BVVGGZB7UEPQ34O3GOBAZWGKV3EHRXOUWTCW5VZEVX25O6DTHGA, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.267237624 (574h17m42.138320669s) [noise=true , address=145.79.227.6:4257 , piece_id=NYWY64OU74DCXARWKJPT3ZA2BNCXRPHGBJSPWDMP7D4Q7JTQATKA, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.267747886 (574h17m42.138830931s) [noise=true , address=45.91.116.5:4300 , piece_id=4CM6TKXWVX774SOVKAOEDCOKPWRASOIKKKH7D2DAVNZRVBDOZ4XQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.268183625 (574h17m42.13926667s) [noise=true , address=145.79.227.47:4266 , piece_id=ROVU66NTSMFVA7NQL3EPYQJPC3NVY6C5MWE72SUVJBOBZ3LXVW4Q, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.268644713 (574h17m42.139727758s) [noise=true , address=77.90.31.14:4661 , piece_id=JB7AXPW64ZN2LE2FINC2IHWAS3IP3XFJUZQGT6VLDDOCQW3WYVKQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.269113045 (574h17m42.14019608s) [noise=true , address=145.79.227.24:4582 , piece_id=GKNT2KXL6SQFX7Q3SU3VEPHXAHDZLT5KRRJV7PH5TEDZWKUFCXVQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.270842227 (574h17m42.141925272s) [noise=true , address=145.79.227.51:4410 , piece_id=JSFOMJHLKR44PYT6TYZKNZRXNYDH2VG6PFPS6JRKHLL5DWI5RO7Q, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.272531123 (574h17m42.143614168s) [noise=true , address=145.79.227.15:4876 , piece_id=2RDPDOOMDXRYFBZUGH34HFNIS2F7UFX324A75AHT56TMEJZ3SSRA, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.274272068 (574h17m42.145355113s) [noise=true , address=77.90.31.21:4877 , piece_id=H2EQIMAN4VLFU5IMGHVGIOAP7MHE57PUMXME5TD3JHVLKBVZOWRA, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.275375680 (574h17m42.146458715s) [noise=true , address=145.79.227.46:4601 , piece_id=7VUAD2MLKJVMNHBLPFETZIFNW26KKSZMHZZADFNB6FYMOWGJIW7Q, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.276772730 (574h17m42.147855775s) [noise=true , address=145.79.227.39:4014 , piece_id=Q2ETXCPCQWDBAHMF2RQR44ALJVSEW7SFUT6XPCJSFPS5L6KYKSFQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.279175966 (574h17m42.150259011s) [noise=true , address=145.79.227.35:4483 , piece_id=JWZHIIEYVO4EY64AJCN7NIH5A7YEI5JNOWRNGR5NJBKTM2ZS6GBQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.280932110 (574h17m42.152015145s) [noise=true , address=45.91.116.9:4262 , piece_id=DUZDADDJZVGT6OO3FWS3H5WMCUZP7OULHDC3BHXIR33UPCTICOJQ, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.284340871 (574h17m42.155423906s) [noise=true , address=145.79.227.37:4567 , piece_id=PGUTNO2ONMCKBXIO5SEQZKMUSORJZVR6OMYPHPNNF67Z6AY4ROQA, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:42.285369390 (574h17m42.156452425s) [noise=true , address=45.91.116.10:4081 , piece_id=UNVNZIH7L5OPQH4S64CXLE4Y2WDQGQVXJKBHDQQFJPCNK5FYTR7A, seg_pos={0 0}, upload=503477]: Uploading piece...
Dec 29 13:46:43.974947354 (574h17m43.846030389s) [upload=503477]: Next returned nil segment
Dec 29 13:46:43.974964145 (574h17m43.84604719s) [upload=503477]: Waiting for error group managing segments...
Dec 29 13:46:44.262745173 (574h17m44.133828208s) [noise=true , address=145.79.227.24:4582 , piece_id=GKNT2KXL6SQFX7Q3SU3VEPHXAHDZLT5KRRJV7PH5TEDZWKUFCXVQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.263074551 (574h17m44.134157586s) [noise=true , address=145.79.227.26:4655 , piece_id=6BVVGGZB7UEPQ34O3GOBAZWGKV3EHRXOUWTCW5VZEVX25O6DTHGA, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.263237851 (574h17m44.134320886s) [noise=true , address=145.79.227.6:4257 , piece_id=NYWY64OU74DCXARWKJPT3ZA2BNCXRPHGBJSPWDMP7D4Q7JTQATKA, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.264133056 (574h17m44.135216091s) [noise=true , address=145.79.227.46:4601 , piece_id=7VUAD2MLKJVMNHBLPFETZIFNW26KKSZMHZZADFNB6FYMOWGJIW7Q, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.269570800 (574h17m44.140653835s) [noise=true , address=145.79.227.15:4876 , piece_id=2RDPDOOMDXRYFBZUGH34HFNIS2F7UFX324A75AHT56TMEJZ3SSRA, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.277117431 (574h17m44.148200476s) [noise=true , address=145.79.227.47:4266 , piece_id=ROVU66NTSMFVA7NQL3EPYQJPC3NVY6C5MWE72SUVJBOBZ3LXVW4Q, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.288278282 (574h17m44.159361317s) [noise=true , address=145.79.227.37:4567 , piece_id=PGUTNO2ONMCKBXIO5SEQZKMUSORJZVR6OMYPHPNNF67Z6AY4ROQA, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.289397734 (574h17m44.160480769s) [noise=true , address=145.79.227.39:4014 , piece_id=Q2ETXCPCQWDBAHMF2RQR44ALJVSEW7SFUT6XPCJSFPS5L6KYKSFQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.293339950 (574h17m44.164422985s) [noise=true , address=145.79.227.41:4086 , piece_id=T5OLI7E4ZPBB63EHXRYA76DRHOHZYVOYH37VCMBCJEUI26JZGANQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.294682116 (574h17m44.165765161s) [noise=true , address=145.79.227.51:4410 , piece_id=JSFOMJHLKR44PYT6TYZKNZRXNYDH2VG6PFPS6JRKHLL5DWI5RO7Q, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:44.632322783 (574h17m44.503405828s) [noise=true , address=45.91.116.9:4262 , piece_id=DUZDADDJZVGT6OO3FWS3H5WMCUZP7OULHDC3BHXIR33UPCTICOJQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:45.239931880 (574h17m45.111014925s) [noise=true , address=77.90.31.14:4661 , piece_id=JB7AXPW64ZN2LE2FINC2IHWAS3IP3XFJUZQGT6VLDDOCQW3WYVKQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:45.303280325 (574h17m45.17436337s) [noise=true , address=45.91.116.5:4300 , piece_id=4CM6TKXWVX774SOVKAOEDCOKPWRASOIKKKH7D2DAVNZRVBDOZ4XQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: <nil>
Dec 29 13:46:45.303302267 (574h17m45.174385302s) [seg_pos={0 0}, upload=503477]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:46:45.303491106 (574h17m45.174574141s) [noise=true , address=145.79.227.35:4483 , piece_id=JWZHIIEYVO4EY64AJCN7NIH5A7YEI5JNOWRNGR5NJBKTM2ZS6GBQ, seg_pos={0 0}, upload=503477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 13:46:45.303498431 (574h17m45.174581476s) [noise=true , address=145.79.227.35:4483 , piece_id=JWZHIIEYVO4EY64AJCN7NIH5A7YEI5JNOWRNGR5NJBKTM2ZS6GBQ, seg_pos={0 0}, upload=503477]: Long tail cancellation due to optimal threshold
Dec 29 13:46:45.303553866 (574h17m45.174636901s) [noise=true , address=45.91.116.10:4081 , piece_id=UNVNZIH7L5OPQH4S64CXLE4Y2WDQGQVXJKBHDQQFJPCNK5FYTR7A, seg_pos={0 0}, upload=503477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 29 13:46:45.303561550 (574h17m45.174644585s) [noise=true , address=45.91.116.10:4081 , piece_id=UNVNZIH7L5OPQH4S64CXLE4Y2WDQGQVXJKBHDQQFJPCNK5FYTR7A, seg_pos={0 0}, upload=503477]: Long tail cancellation due to optimal threshold
Dec 29 13:46:45.303809263 (574h17m45.174892298s) [noise=true , address=77.90.31.21:4877 , piece_id=H2EQIMAN4VLFU5IMGHVGIOAP7MHE57PUMXME5TD3JHVLKBVZOWRA, seg_pos={0 0}, upload=503477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled
Dec 29 13:46:45.303816446 (574h17m45.174899491s) [noise=true , address=77.90.31.21:4877 , piece_id=H2EQIMAN4VLFU5IMGHVGIOAP7MHE57PUMXME5TD3JHVLKBVZOWRA, seg_pos={0 0}, upload=503477]: Long tail cancellation due to optimal threshold
Dec 29 13:46:45.303832216 (574h17m45.174915251s) [seg_pos={0 0}, upload=503477]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:46:45.303899604 (574h17m45.174982639s) [upload=503477]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:46:45.303904223 (574h17m45.174987258s) [upload=503477]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:46:45.594329174 (574h17m45.465412229s) [upload=503477]: Done uploading segments...
Dec 29 13:46:45.594352328 (574h17m45.465435373s) [upload=503477]: Upload finished. err: <nil>
Duration: 30.004878036s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...X.X....X.X....................................X...X.X.XX.] duration:
28.707518463s trace:
https://jaeger.dev.storj.io/trace/21f7513ff9b7e8e9
FAIL (2025-12-29 13:47:11 UTC)
orbiter warning: context deadline exceeded
Duration: 28.707681874s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
54.430204958s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:47:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 54.430220106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
40.505600435s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:46:55 UTC)
orbiter warning: context deadline exceeded
Duration: 40.505633548s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XX] duration:
5.000846083s trace:
https://jaeger.dev.storj.io/trace/3c64c45c31ab9b2d
FAIL (2025-12-29 13:46:42 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000874598s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
53.130776218s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:46:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 53.130806555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [....X.X....X.X....................................X...X.X.XX] duration:
30.001502012s trace:
https://jaeger.dev.storj.io/trace/21f7513ff9b7e8e9
FAIL (2025-12-29 13:46:07 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001524044s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX.] duration:
5.957900547s trace:
https://jaeger.dev.storj.io/trace/6785fab1531ebd87
FAIL (2025-12-29 13:45:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA) context deadline exceeded
======LOGS======
Dec 29 13:45:18.509752331 (574h16m18.380835376s) [upload=503456]: Starting upload
Dec 29 13:45:18.509862981 (574h16m18.380946027s) [upload=503456]: Done starting upload
Dec 29 13:45:18.510057852 (574h16m18.381140897s) [upload=503456]: Uploading segments...
Dec 29 13:45:18.510283893 (574h16m18.381366928s) [upload=503456]: Got next segment. Inline: false
Dec 29 13:45:18.510363434 (574h16m18.381446489s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:45:18.510371750 (574h16m18.381454795s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:45:18.755551037 (574h16m18.626634092s) [seg_pos={0 0}, upload=503456]: Begin upload segment...
Dec 29 13:45:18.755612184 (574h16m18.626695229s) [seg_pos={0 0}, upload=503456]: Done begin upload segment.
Dec 29 13:45:18.755652581 (574h16m18.626735626s) [noise=true , address=145.79.227.38:4978 , piece_id=KA4UFGNLPKFH6UVKW5IW7EOD7K23JP6TLQPNFWM7A65PLR2I76LA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.755701323 (574h16m18.626784368s) [noise=true , address=145.79.227.14:4837 , piece_id=LRJJFX6OIKRBFW7SKH3BZDVYRJO3BFBPNBEZ2WH2JO6CERBHJ4FQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.756198971 (574h16m18.627282016s) [noise=true , address=45.91.116.11:4117 , piece_id=MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.756658877 (574h16m18.627741912s) [noise=true , address=77.90.31.12:4589 , piece_id=T7BPUEGY2HSZ5LY4SDSVSWBCMA4SGJFNKO3DXJAVJWNHC2WFT5PQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.757184397 (574h16m18.628267442s) [noise=true , address=45.91.116.6:4973 , piece_id=Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.757689820 (574h16m18.628772865s) [noise=true , address=145.79.227.37:4567 , piece_id=SDMJIV7Z6T43ZK7WAXTOKTPNRDVYDK6IOX3ZS4WKP735PMUX6TYQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.758336671 (574h16m18.629419716s) [noise=true , address=145.79.227.36:4519 , piece_id=EV535EVJRAQU44RIR6SWUCKZXCL6PTMQ4IBUEYPJXS4VV3HAH7UQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.758793521 (574h16m18.629876566s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.759328880 (574h16m18.630411915s) [noise=true , address=145.79.227.15:4873 , piece_id=F5ITDYGZPMVCNNPE26U4FU3DXNY34QUZ7T2IYS7R2FSWJGG35S7A, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.760704600 (574h16m18.631787645s) [noise=true , address=45.91.116.8:4047 , piece_id=M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.762821470 (574h16m18.633904515s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.765135016 (574h16m18.636218061s) [noise=true , address=45.91.116.5:4299 , piece_id=HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.766590909 (574h16m18.637673954s) [noise=true , address=77.90.31.21:4877 , piece_id=5YGX33VERS4TM4HWKVBQ77OUHVOQVBFFKCNCEKVIRJS6WH64USXA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.768213398 (574h16m18.639296443s) [noise=true , address=145.79.227.48:4301 , piece_id=VAJKXL7OZGO2ZRRE6ETAJCHZFE6RG7YLRY4NQAMPWQQIITOX3MXQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.770056097 (574h16m18.641139132s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.771744943 (574h16m18.642827988s) [noise=true , address=145.79.227.2:4114 , piece_id=TU6ZPDZAKT4W4QZUXRS5B2Z2MP2ZD5R6DU2A2HXKNHYCANYMKMJA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:20.376379563 (574h16m20.247462608s) [upload=503456]: Next returned nil segment
Dec 29 13:45:20.376397317 (574h16m20.247480362s) [upload=503456]: Waiting for error group managing segments...
Dec 29 13:45:20.748746595 (574h16m20.61982964s) [noise=true , address=145.79.227.36:4519 , piece_id=EV535EVJRAQU44RIR6SWUCKZXCL6PTMQ4IBUEYPJXS4VV3HAH7UQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.751335374 (574h16m20.622418419s) [noise=true , address=145.79.227.37:4567 , piece_id=SDMJIV7Z6T43ZK7WAXTOKTPNRDVYDK6IOX3ZS4WKP735PMUX6TYQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.755472672 (574h16m20.626555717s) [noise=true , address=145.79.227.15:4873 , piece_id=F5ITDYGZPMVCNNPE26U4FU3DXNY34QUZ7T2IYS7R2FSWJGG35S7A, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.768386462 (574h16m20.639469507s) [noise=true , address=145.79.227.2:4114 , piece_id=TU6ZPDZAKT4W4QZUXRS5B2Z2MP2ZD5R6DU2A2HXKNHYCANYMKMJA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.770907071 (574h16m20.641990126s) [noise=true , address=145.79.227.14:4837 , piece_id=LRJJFX6OIKRBFW7SKH3BZDVYRJO3BFBPNBEZ2WH2JO6CERBHJ4FQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.770993696 (574h16m20.642076741s) [noise=true , address=145.79.227.38:4978 , piece_id=KA4UFGNLPKFH6UVKW5IW7EOD7K23JP6TLQPNFWM7A65PLR2I76LA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.776453973 (574h16m20.647537008s) [noise=true , address=145.79.227.48:4301 , piece_id=VAJKXL7OZGO2ZRRE6ETAJCHZFE6RG7YLRY4NQAMPWQQIITOX3MXQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.014542770 (574h16m20.885625805s) [noise=true , address=77.90.31.21:4877 , piece_id=5YGX33VERS4TM4HWKVBQ77OUHVOQVBFFKCNCEKVIRJS6WH64USXA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.023183904 (574h16m20.894266949s) [noise=true , address=45.91.116.8:4047 , piece_id=M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.023674408 (574h16m20.894757453s) [noise=true , address=45.91.116.5:4299 , piece_id=HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.083805846 (574h16m20.954888881s) [noise=true , address=77.90.31.12:4589 , piece_id=T7BPUEGY2HSZ5LY4SDSVSWBCMA4SGJFNKO3DXJAVJWNHC2WFT5PQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.084795251 (574h16m20.955878296s) [noise=true , address=45.91.116.11:4117 , piece_id=MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.149117049 (574h16m21.020200094s) [noise=true , address=45.91.116.6:4973 , piece_id=Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.149139552 (574h16m21.020222597s) [seg_pos={0 0}, upload=503456]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:45:21.149389708 (574h16m21.020472743s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.149398344 (574h16m21.020481389s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.149771575 (574h16m21.02085461s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.149779981 (574h16m21.020863016s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.150013456 (574h16m21.021096501s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.150022062 (574h16m21.021105097s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.150041199 (574h16m21.021124234s) [seg_pos={0 0}, upload=503456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:45:21.150193899 (574h16m21.021276944s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:45:21.150201433 (574h16m21.021284478s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:45:21.447744209 (574h16m21.318827274s) [upload=503456]: Done uploading segments...
Dec 29 13:45:21.447774717 (574h16m21.318857792s) [upload=503456]: Upload finished. err: <nil>
Duration: 5.95792872s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XXX] duration:
30.003475594s trace:
https://jaeger.dev.storj.io/trace/6785fab1531ebd87
FAIL (2025-12-29 13:45:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA) context deadline exceeded
======LOGS======
Dec 29 13:45:18.509752331 (574h16m18.380835376s) [upload=503456]: Starting upload
Dec 29 13:45:18.509862981 (574h16m18.380946027s) [upload=503456]: Done starting upload
Dec 29 13:45:18.510057852 (574h16m18.381140897s) [upload=503456]: Uploading segments...
Dec 29 13:45:18.510283893 (574h16m18.381366928s) [upload=503456]: Got next segment. Inline: false
Dec 29 13:45:18.510363434 (574h16m18.381446489s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:45:18.510371750 (574h16m18.381454795s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:45:18.755551037 (574h16m18.626634092s) [seg_pos={0 0}, upload=503456]: Begin upload segment...
Dec 29 13:45:18.755612184 (574h16m18.626695229s) [seg_pos={0 0}, upload=503456]: Done begin upload segment.
Dec 29 13:45:18.755652581 (574h16m18.626735626s) [noise=true , address=145.79.227.38:4978 , piece_id=KA4UFGNLPKFH6UVKW5IW7EOD7K23JP6TLQPNFWM7A65PLR2I76LA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.755701323 (574h16m18.626784368s) [noise=true , address=145.79.227.14:4837 , piece_id=LRJJFX6OIKRBFW7SKH3BZDVYRJO3BFBPNBEZ2WH2JO6CERBHJ4FQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.756198971 (574h16m18.627282016s) [noise=true , address=45.91.116.11:4117 , piece_id=MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.756658877 (574h16m18.627741912s) [noise=true , address=77.90.31.12:4589 , piece_id=T7BPUEGY2HSZ5LY4SDSVSWBCMA4SGJFNKO3DXJAVJWNHC2WFT5PQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.757184397 (574h16m18.628267442s) [noise=true , address=45.91.116.6:4973 , piece_id=Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.757689820 (574h16m18.628772865s) [noise=true , address=145.79.227.37:4567 , piece_id=SDMJIV7Z6T43ZK7WAXTOKTPNRDVYDK6IOX3ZS4WKP735PMUX6TYQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.758336671 (574h16m18.629419716s) [noise=true , address=145.79.227.36:4519 , piece_id=EV535EVJRAQU44RIR6SWUCKZXCL6PTMQ4IBUEYPJXS4VV3HAH7UQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.758793521 (574h16m18.629876566s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.759328880 (574h16m18.630411915s) [noise=true , address=145.79.227.15:4873 , piece_id=F5ITDYGZPMVCNNPE26U4FU3DXNY34QUZ7T2IYS7R2FSWJGG35S7A, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.760704600 (574h16m18.631787645s) [noise=true , address=45.91.116.8:4047 , piece_id=M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.762821470 (574h16m18.633904515s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.765135016 (574h16m18.636218061s) [noise=true , address=45.91.116.5:4299 , piece_id=HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.766590909 (574h16m18.637673954s) [noise=true , address=77.90.31.21:4877 , piece_id=5YGX33VERS4TM4HWKVBQ77OUHVOQVBFFKCNCEKVIRJS6WH64USXA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.768213398 (574h16m18.639296443s) [noise=true , address=145.79.227.48:4301 , piece_id=VAJKXL7OZGO2ZRRE6ETAJCHZFE6RG7YLRY4NQAMPWQQIITOX3MXQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.770056097 (574h16m18.641139132s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:18.771744943 (574h16m18.642827988s) [noise=true , address=145.79.227.2:4114 , piece_id=TU6ZPDZAKT4W4QZUXRS5B2Z2MP2ZD5R6DU2A2HXKNHYCANYMKMJA, seg_pos={0 0}, upload=503456]: Uploading piece...
Dec 29 13:45:20.376379563 (574h16m20.247462608s) [upload=503456]: Next returned nil segment
Dec 29 13:45:20.376397317 (574h16m20.247480362s) [upload=503456]: Waiting for error group managing segments...
Dec 29 13:45:20.748746595 (574h16m20.61982964s) [noise=true , address=145.79.227.36:4519 , piece_id=EV535EVJRAQU44RIR6SWUCKZXCL6PTMQ4IBUEYPJXS4VV3HAH7UQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.751335374 (574h16m20.622418419s) [noise=true , address=145.79.227.37:4567 , piece_id=SDMJIV7Z6T43ZK7WAXTOKTPNRDVYDK6IOX3ZS4WKP735PMUX6TYQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.755472672 (574h16m20.626555717s) [noise=true , address=145.79.227.15:4873 , piece_id=F5ITDYGZPMVCNNPE26U4FU3DXNY34QUZ7T2IYS7R2FSWJGG35S7A, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.768386462 (574h16m20.639469507s) [noise=true , address=145.79.227.2:4114 , piece_id=TU6ZPDZAKT4W4QZUXRS5B2Z2MP2ZD5R6DU2A2HXKNHYCANYMKMJA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.770907071 (574h16m20.641990126s) [noise=true , address=145.79.227.14:4837 , piece_id=LRJJFX6OIKRBFW7SKH3BZDVYRJO3BFBPNBEZ2WH2JO6CERBHJ4FQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.770993696 (574h16m20.642076741s) [noise=true , address=145.79.227.38:4978 , piece_id=KA4UFGNLPKFH6UVKW5IW7EOD7K23JP6TLQPNFWM7A65PLR2I76LA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:20.776453973 (574h16m20.647537008s) [noise=true , address=145.79.227.48:4301 , piece_id=VAJKXL7OZGO2ZRRE6ETAJCHZFE6RG7YLRY4NQAMPWQQIITOX3MXQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.014542770 (574h16m20.885625805s) [noise=true , address=77.90.31.21:4877 , piece_id=5YGX33VERS4TM4HWKVBQ77OUHVOQVBFFKCNCEKVIRJS6WH64USXA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.023183904 (574h16m20.894266949s) [noise=true , address=45.91.116.8:4047 , piece_id=M2NZUTYX67M3TP6VUGQKPFZ4BS432ZCBYY4WRMSAHXU4LAM552UA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.023674408 (574h16m20.894757453s) [noise=true , address=45.91.116.5:4299 , piece_id=HI3ZNICR7UFH2XLBLEJ5TN4URFCZJUGH2CAYBYMG6JXMMQDISEGA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.083805846 (574h16m20.954888881s) [noise=true , address=77.90.31.12:4589 , piece_id=T7BPUEGY2HSZ5LY4SDSVSWBCMA4SGJFNKO3DXJAVJWNHC2WFT5PQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.084795251 (574h16m20.955878296s) [noise=true , address=45.91.116.11:4117 , piece_id=MDL4JE4W4QRUS2TK7KE67YQWRFYUMOKEWYHO5QUQGKQC7XEVHZVA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.149117049 (574h16m21.020200094s) [noise=true , address=45.91.116.6:4973 , piece_id=Z5W4SNLGO5VQPK7EMAM42PZ4PNG2UN6KEE37LFFJQ3J75NFTN44Q, seg_pos={0 0}, upload=503456]: Done uploading piece. err: <nil>
Dec 29 13:45:21.149139552 (574h16m21.020222597s) [seg_pos={0 0}, upload=503456]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:45:21.149389708 (574h16m21.020472743s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.149398344 (574h16m21.020481389s) [noise=true , address=145.79.227.34:4907 , piece_id=UUWWQZ62AO46PJNSK6VZ7TRTV7RTW7FG7B4NX5MGZWID4TBXTV5Q, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.149771575 (574h16m21.02085461s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.149779981 (574h16m21.020863016s) [noise=true , address=145.79.227.49:4339 , piece_id=ALWTB37CAOSQ66UERNVHXJ2PYBPCMSBLPJDTVOQQR7JVA6QCDATQ, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.150013456 (574h16m21.021096501s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 29 13:45:21.150022062 (574h16m21.021105097s) [noise=true , address=45.91.116.4:4939 , piece_id=ZKHPMDCBY23Q6DGV2FTDHF6Z5BX5JNWS4UFCUYRKNV4O4KWDGEZA, seg_pos={0 0}, upload=503456]: Long tail cancellation due to optimal threshold
Dec 29 13:45:21.150041199 (574h16m21.021124234s) [seg_pos={0 0}, upload=503456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:45:21.150193899 (574h16m21.021276944s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:45:21.150201433 (574h16m21.021284478s) [upload=503456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:45:21.447744209 (574h16m21.318827274s) [upload=503456]: Done uploading segments...
Dec 29 13:45:21.447774717 (574h16m21.318857792s) [upload=503456]: Upload finished. err: <nil>
Duration: 30.003496364s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.X] duration:
5.001663534s trace:
https://jaeger.dev.storj.io/trace/4b0d318e945029e8
FAIL (2025-12-29 13:45:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00169324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
15.840909112s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:45:23 UTC)
orbiter warning: context deadline exceeded
Duration: 15.840936674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
56.322231698s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:45:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 56.322257417s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.XX] duration:
30.003932356s trace:
https://jaeger.dev.storj.io/trace/515964eddf04cbe2
FAIL (2025-12-29 13:45:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 2SZQLMB3ICAKYNTEK2LSAG3HM6KZLUSQPPXOXNBSK2HD5SZ4UUEQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SHAI6HGEA3LJS7XQS7PMRQLYBJMLOM3JZ3NEROXZPYPJLOB2JUKA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 4D2L33TIDLAYDESL37QPH6GKPVVWSZEDSO2RCIX4RSCSWAK6MURA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: YLZ4P4YT5PX5ZEKBAZC4D74K6XCXA6CA54IWEW72AB3I4GWBMSNA) context deadline exceeded
======LOGS======
Dec 29 13:44:42.205937669 (574h15m42.077020704s) [upload=503448]: Starting upload
Dec 29 13:44:42.206031007 (574h15m42.077114042s) [upload=503448]: Done starting upload
Dec 29 13:44:42.206271976 (574h15m42.077355031s) [upload=503448]: Uploading segments...
Dec 29 13:44:42.206460324 (574h15m42.077543359s) [upload=503448]: Got next segment. Inline: false
Dec 29 13:44:42.206497775 (574h15m42.07758081s) [upload=503448]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:44:42.206503257 (574h15m42.077586292s) [upload=503448]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:44:42.454784610 (574h15m42.325867655s) [seg_pos={0 0}, upload=503448]: Begin upload segment...
Dec 29 13:44:42.454832441 (574h15m42.325915486s) [seg_pos={0 0}, upload=503448]: Done begin upload segment.
Dec 29 13:44:42.454857669 (574h15m42.325940704s) [noise=true , address=45.91.116.5:4298 , piece_id=SHAI6HGEA3LJS7XQS7PMRQLYBJMLOM3JZ3NEROXZPYPJLOB2JUKA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.455011772 (574h15m42.326094807s) [noise=true , address=145.79.227.51:4409 , piece_id=PVKSLK3TDF27Y6KE4VD37K46ENWY6FCFSMF3CMHNDBZLRNR3DFMQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.455430099 (574h15m42.326513144s) [noise=true , address=45.91.116.4:4937 , piece_id=MSKAPVNM64T5R5KGS4PFZNQWYWBCY2IHAMUD4O6722U43I6VQ5HA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.455902178 (574h15m42.326985213s) [noise=true , address=45.91.116.6:4973 , piece_id=4D2L33TIDLAYDESL37QPH6GKPVVWSZEDSO2RCIX4RSCSWAK6MURA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.456331174 (574h15m42.32741422s) [noise=true , address=145.79.227.19:4078 , piece_id=QTZUQDOXCXFN3ZHG2NUARQF7AWWHNSW5V6O7FQPN3L733YW6KSMQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.456745123 (574h15m42.327828158s) [noise=true , address=145.79.227.6:4259 , piece_id=BKJFJD6APM4P3625ZF2BO6B7VCFCB6C4W2QBLLJWQO55WBCVVZOQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457021098 (574h15m42.328104133s) [noise=true , address=145.79.227.46:4603 , piece_id=52MGKPFQ62KPEV5UH4ERJITLF666CNJ4N7ENW5S7EVP65A73XCZQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457052218 (574h15m42.328135263s) [noise=true , address=145.79.227.40:4050 , piece_id=NSDR7ZZQQAV3NORW5GFI757FOSHEVZAKRRIT2AQWH57ADWYR3USQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457285643 (574h15m42.328368688s) [noise=true , address=145.79.227.52:4446 , piece_id=O2PORHCPMQD65KONPXS3FKTVMVNJE3HYXEJWIYMYP26GGOVTPZOQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457458792 (574h15m42.328541827s) [noise=true , address=45.91.116.12:4226 , piece_id=YLZ4P4YT5PX5ZEKBAZC4D74K6XCXA6CA54IWEW72AB3I4GWBMSNA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457554554 (574h15m42.328637589s) [noise=true , address=45.91.116.7:4011 , piece_id=NG6VWPX5LUYMGLAKJ7I5OTWZCCLEUG742DS2KRUCEAPY75XTZYMQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.457852492 (574h15m42.328935537s) [noise=true , address=77.90.31.13:4626 , piece_id=TPOUOXXGWDEVUVPBUIPJ23BXJ4LVKF74ZMB5VP5V66ZRZTUXYRFA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.458448066 (574h15m42.329531101s) [noise=true , address=45.91.116.13:4153 , piece_id=2SZQLMB3ICAKYNTEK2LSAG3HM6KZLUSQPPXOXNBSK2HD5SZ4UUEQ, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.458830213 (574h15m42.329913258s) [noise=true , address=45.91.116.11:4119 , piece_id=SSODHJTT4QPGM4BI2VHMFYMTJMVT5EPCXFNLSMVMZXGO37BGTVIA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.462967572 (574h15m42.334050607s) [noise=true , address=145.79.227.20:4437 , piece_id=6CEH5XU5FHJYDDQOSAJAHSHQPNZMXLDUZAUXY7QHL5NAMTBZK2MA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:42.464192754 (574h15m42.335275789s) [noise=true , address=145.79.227.22:4511 , piece_id=Q6O67YQ3O5ETUK5DC74NZPVMCASUB7VZRDGY5MXWKDIHIDJ4WLAA, seg_pos={0 0}, upload=503448]: Uploading piece...
Dec 29 13:44:44.067282273 (574h15m43.938365308s) [upload=503448]: Next returned nil segment
Dec 29 13:44:44.067296319 (574h15m43.938379365s) [upload=503448]: Waiting for error group managing segments...
Dec 29 13:44:44.445790518 (574h15m44.316873553s) [noise=true , address=145.79.227.46:4603 , piece_id=52MGKPFQ62KPEV5UH4ERJITLF666CNJ4N7ENW5S7EVP65A73XCZQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.447357462 (574h15m44.318440497s) [noise=true , address=145.79.227.6:4259 , piece_id=BKJFJD6APM4P3625ZF2BO6B7VCFCB6C4W2QBLLJWQO55WBCVVZOQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.451286954 (574h15m44.322369989s) [noise=true , address=145.79.227.52:4446 , piece_id=O2PORHCPMQD65KONPXS3FKTVMVNJE3HYXEJWIYMYP26GGOVTPZOQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.458066814 (574h15m44.329149849s) [noise=true , address=145.79.227.19:4078 , piece_id=QTZUQDOXCXFN3ZHG2NUARQF7AWWHNSW5V6O7FQPN3L733YW6KSMQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.472734664 (574h15m44.343817709s) [noise=true , address=145.79.227.51:4409 , piece_id=PVKSLK3TDF27Y6KE4VD37K46ENWY6FCFSMF3CMHNDBZLRNR3DFMQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.477790912 (574h15m44.348873957s) [noise=true , address=145.79.227.22:4511 , piece_id=Q6O67YQ3O5ETUK5DC74NZPVMCASUB7VZRDGY5MXWKDIHIDJ4WLAA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.480724057 (574h15m44.351807102s) [noise=true , address=145.79.227.20:4437 , piece_id=6CEH5XU5FHJYDDQOSAJAHSHQPNZMXLDUZAUXY7QHL5NAMTBZK2MA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.694479168 (574h15m44.565562223s) [noise=true , address=45.91.116.5:4298 , piece_id=SHAI6HGEA3LJS7XQS7PMRQLYBJMLOM3JZ3NEROXZPYPJLOB2JUKA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.739727728 (574h15m44.610810763s) [noise=true , address=77.90.31.13:4626 , piece_id=TPOUOXXGWDEVUVPBUIPJ23BXJ4LVKF74ZMB5VP5V66ZRZTUXYRFA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.750915111 (574h15m44.621998156s) [noise=true , address=45.91.116.13:4153 , piece_id=2SZQLMB3ICAKYNTEK2LSAG3HM6KZLUSQPPXOXNBSK2HD5SZ4UUEQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.823646982 (574h15m44.694730017s) [noise=true , address=45.91.116.7:4011 , piece_id=NG6VWPX5LUYMGLAKJ7I5OTWZCCLEUG742DS2KRUCEAPY75XTZYMQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:44.991317001 (574h15m44.862400046s) [noise=true , address=45.91.116.6:4973 , piece_id=4D2L33TIDLAYDESL37QPH6GKPVVWSZEDSO2RCIX4RSCSWAK6MURA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:45.179018942 (574h15m45.050101977s) [noise=true , address=45.91.116.12:4226 , piece_id=YLZ4P4YT5PX5ZEKBAZC4D74K6XCXA6CA54IWEW72AB3I4GWBMSNA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: <nil>
Dec 29 13:44:45.179037438 (574h15m45.050120473s) [seg_pos={0 0}, upload=503448]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:44:45.179221598 (574h15m45.050304633s) [noise=true , address=45.91.116.11:4119 , piece_id=SSODHJTT4QPGM4BI2VHMFYMTJMVT5EPCXFNLSMVMZXGO37BGTVIA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 29 13:44:45.179227940 (574h15m45.050310975s) [noise=true , address=45.91.116.11:4119 , piece_id=SSODHJTT4QPGM4BI2VHMFYMTJMVT5EPCXFNLSMVMZXGO37BGTVIA, seg_pos={0 0}, upload=503448]: Long tail cancellation due to optimal threshold
Dec 29 13:44:45.179330776 (574h15m45.050413821s) [noise=true , address=45.91.116.4:4937 , piece_id=MSKAPVNM64T5R5KGS4PFZNQWYWBCY2IHAMUD4O6722U43I6VQ5HA, seg_pos={0 0}, upload=503448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 29 13:44:45.179336066 (574h15m45.050419111s) [noise=true , address=45.91.116.4:4937 , piece_id=MSKAPVNM64T5R5KGS4PFZNQWYWBCY2IHAMUD4O6722U43I6VQ5HA, seg_pos={0 0}, upload=503448]: Long tail cancellation due to optimal threshold
Dec 29 13:44:45.179534183 (574h15m45.050617229s) [noise=true , address=145.79.227.40:4050 , piece_id=NSDR7ZZQQAV3NORW5GFI757FOSHEVZAKRRIT2AQWH57ADWYR3USQ, seg_pos={0 0}, upload=503448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 29 13:44:45.179549432 (574h15m45.050632467s) [noise=true , address=145.79.227.40:4050 , piece_id=NSDR7ZZQQAV3NORW5GFI757FOSHEVZAKRRIT2AQWH57ADWYR3USQ, seg_pos={0 0}, upload=503448]: Long tail cancellation due to optimal threshold
Dec 29 13:44:45.179571304 (574h15m45.050654339s) [seg_pos={0 0}, upload=503448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:44:45.179712052 (574h15m45.050795097s) [upload=503448]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:44:45.179719025 (574h15m45.05080206s) [upload=503448]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:44:45.472232404 (574h15m45.343315449s) [upload=503448]: Done uploading segments...
Dec 29 13:44:45.472255108 (574h15m45.343338143s) [upload=503448]: Upload finished. err: <nil>
Duration: 30.003956293s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [X....X.X....X.X....................................X...X.X.X] duration:
30.001903423s trace:
https://jaeger.dev.storj.io/trace/5cdca3c39cf83d
FAIL (2025-12-29 13:45:04 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001936394s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [X.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.] duration:
3.618861614s trace:
https://jaeger.dev.storj.io/trace/b61e4de0edcaa6f
FAIL (2025-12-29 13:44:57 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.618890638s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
17.291702651s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:44:57 UTC)
orbiter warning: context deadline exceeded
Duration: 17.291716648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
19.323377392s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:44:40 UTC)
orbiter warning: context deadline exceeded
Duration: 19.323403421s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [X.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.X.X] duration:
30.003966308s trace:
https://jaeger.dev.storj.io/trace/2a848d49d569a56f
FAIL (2025-12-29 13:44:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4 (574h14m47.689456229s) [upload=503436]: Done starting upload
Dec 29 13:43:47.818442887 (574h14m47.689525932s) [upload=503436]: Uploading segments...
Dec 29 13:43:47.818640003 (574h14m47.689723048s) [upload=503436]: Got next segment. Inline: false
Dec 29 13:43:47.818704225 (574h14m47.68978726s) [upload=503436]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:43:47.818709525 (574h14m47.68979257s) [upload=503436]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:43:48.068018677 (574h14m47.939101712s) [seg_pos={0 0}, upload=503436]: Begin upload segment...
Dec 29 13:43:48.068095293 (574h14m47.939178338s) [seg_pos={0 0}, upload=503436]: Done begin upload segment.
Dec 29 13:43:48.068150006 (574h14m47.939233041s) [noise=true , address=145.79.227.12:4767 , piece_id=H2FMV5YD7GSE73362YJQ5J5WZZI6SOA6AOVZWSB6JRRCGL35I7PQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.068189472 (574h14m47.939272517s) [noise=true , address=145.79.227.42:4122 , piece_id=JZ73DWDUDMTXB2NACF3MFLNA2UYCORSI5HMT2GVHNSDKMQKQUNJA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.068860971 (574h14m47.939944016s) [noise=true , address=45.91.116.6:4975 , piece_id=EYPSJ2GDFK2DB5BLFQAQC6OZZ6A7IRMTUNHTCH3774ILQDHFZ6KA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.069272785 (574h14m47.94035582s) [noise=true , address=145.79.227.50:4375 , piece_id=U7VOS6BJNCL7F74FSYXCJQ2OC5OQRBUUM23DBHSRF7XC2D6IBI7A, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.069691723 (574h14m47.940774758s) [noise=true , address=45.91.116.7:4011 , piece_id=LXQZVRAQJWFJKDOFUQRQGQG6OIPLUYJSUYZZT3SRNL3IQER347PQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.070270805 (574h14m47.94135384s) [noise=true , address=145.79.227.37:4567 , piece_id=XQWOILPXIGSR3R256UGQ3W6NGVZ2OFZHAIDLE34ACRVT5HAU6KYQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.070657080 (574h14m47.941740125s) [noise=true , address=145.79.227.13:4803 , piece_id=C2IXXXGASDRKYZJK3V3EVIKMA7BJA4W7VML25242RQ6CLDLY3YPQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.071017317 (574h14m47.942100352s) [noise=true , address=145.79.227.52:4445 , piece_id=S3CRXHM4TTCXNTUQIIE3BKGEJARZUPLBA26TJCVARUCCW3OR7W4A, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.071479456 (574h14m47.942562491s) [noise=true , address=145.79.227.14:4840 , piece_id=PKOMJ7XFP7WTYZPCOY6DIO4NVZLTQCW7BPIXPH5JN7PP4CBV7ISQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.071926167 (574h14m47.943009202s) [noise=true , address=145.79.227.38:4979 , piece_id=7DDCI3P75RBVMYOTZKNOXYH6DYHG5HJEAHBHGMF3MAE5WSGSAKOQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.072814729 (574h14m47.943897764s) [noise=true , address=145.79.227.45:4230 , piece_id=QAHQR2UAKZNNWMKW37QHMXWA4KER3GG7LF2T6SRBSKQY3XSBGOAQ, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.073950622 (574h14m47.945033667s) [noise=true , address=145.79.227.32:4834 , piece_id=GZRJBOVBO5KKJ5O6E7KBIQXHW5D3Y2PGZXVUN2HQQHSPKQNGNBEA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.074438260 (574h14m47.945521296s) [noise=true , address=77.90.31.17:4773 , piece_id=PODR2KKQD6UENE4NWNMG6BMCRTHHYPRKKKWWSJ4DEQKTGMIBA74Q, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.075572379 (574h14m47.946655415s) [noise=true , address=77.90.31.15:4697 , piece_id=4MFV75M6CJKGBPDP6J7JDOA3CGYBG25MGOOHWPIOMYLS3LANU6HA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.076496640 (574h14m47.947579675s) [noise=true , address=45.91.116.8:4047 , piece_id=KMOFOBZJSQHOJ7SERDJ3XSEOBTH57DSAFI3LCAQE5UUVD6ORBRAA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:48.076702251 (574h14m47.947785287s) [noise=true , address=145.79.227.46:4601 , piece_id=HE5LUP2WK5P3LX6GKDAVDQIPHU7KRO3NJFPNARH2ZVOXZOSWSUBA, seg_pos={0 0}, upload=503436]: Uploading piece...
Dec 29 13:43:49.769820307 (574h14m49.640903342s) [upload=503436]: Next returned nil segment
Dec 29 13:43:49.769842228 (574h14m49.640925263s) [upload=503436]: Waiting for error group managing segments...
Dec 29 13:43:50.050541241 (574h14m49.921624286s) [noise=true , address=145.79.227.45:4230 , piece_id=QAHQR2UAKZNNWMKW37QHMXWA4KER3GG7LF2T6SRBSKQY3XSBGOAQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.052960989 (574h14m49.924044034s) [noise=true , address=145.79.227.12:4767 , piece_id=H2FMV5YD7GSE73362YJQ5J5WZZI6SOA6AOVZWSB6JRRCGL35I7PQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.065030801 (574h14m49.936113846s) [noise=true , address=145.79.227.13:4803 , piece_id=C2IXXXGASDRKYZJK3V3EVIKMA7BJA4W7VML25242RQ6CLDLY3YPQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.066997866 (574h14m49.938080901s) [noise=true , address=145.79.227.32:4834 , piece_id=GZRJBOVBO5KKJ5O6E7KBIQXHW5D3Y2PGZXVUN2HQQHSPKQNGNBEA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.067403249 (574h14m49.938486294s) [noise=true , address=145.79.227.42:4122 , piece_id=JZ73DWDUDMTXB2NACF3MFLNA2UYCORSI5HMT2GVHNSDKMQKQUNJA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.076739266 (574h14m49.947822301s) [noise=true , address=145.79.227.14:4840 , piece_id=PKOMJ7XFP7WTYZPCOY6DIO4NVZLTQCW7BPIXPH5JN7PP4CBV7ISQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.083795201 (574h14m49.954878246s) [noise=true , address=145.79.227.50:4375 , piece_id=U7VOS6BJNCL7F74FSYXCJQ2OC5OQRBUUM23DBHSRF7XC2D6IBI7A, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.086251458 (574h14m49.957334503s) [noise=true , address=145.79.227.37:4567 , piece_id=XQWOILPXIGSR3R256UGQ3W6NGVZ2OFZHAIDLE34ACRVT5HAU6KYQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.086785375 (574h14m49.95786842s) [noise=true , address=145.79.227.46:4601 , piece_id=HE5LUP2WK5P3LX6GKDAVDQIPHU7KRO3NJFPNARH2ZVOXZOSWSUBA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.090951939 (574h14m49.962034984s) [noise=true , address=145.79.227.38:4979 , piece_id=7DDCI3P75RBVMYOTZKNOXYH6DYHG5HJEAHBHGMF3MAE5WSGSAKOQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.091193258 (574h14m49.962276293s) [noise=true , address=145.79.227.52:4445 , piece_id=S3CRXHM4TTCXNTUQIIE3BKGEJARZUPLBA26TJCVARUCCW3OR7W4A, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.322348812 (574h14m50.193431857s) [noise=true , address=77.90.31.17:4773 , piece_id=PODR2KKQD6UENE4NWNMG6BMCRTHHYPRKKKWWSJ4DEQKTGMIBA74Q, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.762556713 (574h14m50.633639748s) [noise=true , address=45.91.116.6:4975 , piece_id=EYPSJ2GDFK2DB5BLFQAQC6OZZ6A7IRMTUNHTCH3774ILQDHFZ6KA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: <nil>
Dec 29 13:43:50.762576290 (574h14m50.633659325s) [seg_pos={0 0}, upload=503436]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:43:50.762879478 (574h14m50.633962513s) [noise=true , address=45.91.116.7:4011 , piece_id=LXQZVRAQJWFJKDOFUQRQGQG6OIPLUYJSUYZZT3SRNL3IQER347PQ, seg_pos={0 0}, upload=503436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:36208->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:36208->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36208->45.91.116.7:4011: use of closed network connection
Dec 29 13:43:50.762886992 (574h14m50.633970037s) [noise=true , address=45.91.116.7:4011 , piece_id=LXQZVRAQJWFJKDOFUQRQGQG6OIPLUYJSUYZZT3SRNL3IQER347PQ, seg_pos={0 0}, upload=503436]: Long tail cancellation due to optimal threshold
Dec 29 13:43:50.763043540 (574h14m50.634126575s) [noise=true , address=45.91.116.8:4047 , piece_id=KMOFOBZJSQHOJ7SERDJ3XSEOBTH57DSAFI3LCAQE5UUVD6ORBRAA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:50094->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 13:43:50.763049802 (574h14m50.634132847s) [noise=true , address=45.91.116.8:4047 , piece_id=KMOFOBZJSQHOJ7SERDJ3XSEOBTH57DSAFI3LCAQE5UUVD6ORBRAA, seg_pos={0 0}, upload=503436]: Long tail cancellation due to optimal threshold
Dec 29 13:43:50.763112351 (574h14m50.634195396s) [noise=true , address=77.90.31.15:4697 , piece_id=4MFV75M6CJKGBPDP6J7JDOA3CGYBG25MGOOHWPIOMYLS3LANU6HA, seg_pos={0 0}, upload=503436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 29 13:43:50.763129974 (574h14m50.634213009s) [noise=true , address=77.90.31.15:4697 , piece_id=4MFV75M6CJKGBPDP6J7JDOA3CGYBG25MGOOHWPIOMYLS3LANU6HA, seg_pos={0 0}, upload=503436]: Long tail cancellation due to optimal threshold
Dec 29 13:43:50.763156645 (574h14m50.63423969s) [seg_pos={0 0}, upload=503436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:43:50.763306150 (574h14m50.634389195s) [upload=503436]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:43:50.763314125 (574h14m50.63439718s) [upload=503436]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:43:51.055843664 (574h14m50.926926699s) [upload=503436]: Done uploading segments...
Dec 29 13:43:51.055872900 (574h14m50.926955945s) [upload=503436]: Upload finished. err: <nil>
Duration: 30.003991656s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
16.188039713s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:44:06 UTC)
orbiter warning: context deadline exceeded
Duration: 16.188055624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration:
5.001707945s trace:
https://jaeger.dev.storj.io/trace/b61e4de0edcaa6f
FAIL (2025-12-29 13:43:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001747809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.....................XX....................................] duration:
30.544948606s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:43:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 30.544964878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
24.662059882s trace:
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:43:50 UTC)
orbiter warning: context deadline exceeded
Duration: 24.66208478s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X....X.X....X.X....................................X...X.X.] duration:
16.411441476s trace:
https://jaeger.dev.storj.io/trace/231023e77411cc8
FAIL (2025-12-29 13:43:44 UTC)
orbiter warning: context deadline exceeded
Duration: 16.411468026s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X....X.X....X.X....................................X...X.X] duration:
30.001454613s trace:
https://jaeger.dev.storj.io/trace/231023e77411cc8
FAIL (2025-12-29 13:43:25 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001477527s
edge
gateway
us-select-1
|