Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.488252954s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:47:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.488282309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.88721733s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:47:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 5.887244102s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.218553131s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:47:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.218583909s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.738456696s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:46:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 5.738479348s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
13.763070348s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:46:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 13.763100525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.062145139s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:46:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.06217175s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.457956532s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:46:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.457974756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.632958725s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:46:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 5.632986838s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
15.70370504s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:45:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 15.703764535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
5.764805868s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:45:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.764827339s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.656520526s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:45:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.656550944s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.44832947s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:44:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.448359879s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.885433485s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:44:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 5.885464765s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
11.93297207s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:44:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 11.933014581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.599980985s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:44:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.600007536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
26.622404868s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:44:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 26.622429296s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.20178552s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:44:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.201858018s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.781430654s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:44:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 5.781467604s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
6.196800536s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:44:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.196832196s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
3.488884616s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:43:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.48890787s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001802202s trace:
https://jaeger.dev.storj.io/trace/3d17fb12eeb48f64
FAIL (2025-12-16 07:43:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ) context deadline exceeded
======LOGS======
Dec 16 07:43:00.504118065 (256h14m0.3752011s) [upload=224664]: Starting upload
Dec 16 07:43:00.504201053 (256h14m0.375284088s) [upload=224664]: Done starting upload
Dec 16 07:43:00.504286566 (256h14m0.375369631s) [upload=224664]: Uploading segments...
Dec 16 07:43:00.504540349 (256h14m0.375623394s) [upload=224664]: Got next segment. Inline: false
Dec 16 07:43:00.504584433 (256h14m0.375667478s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 16 07:43:00.504591818 (256h14m0.375674853s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 16 07:43:00.754443854 (256h14m0.625526899s) [seg_pos={0 0}, upload=224664]: Begin upload segment...
Dec 16 07:43:00.754498899 (256h14m0.625581944s) [seg_pos={0 0}, upload=224664]: Done begin upload segment.
Dec 16 07:43:00.754544225 (256h14m0.62562727s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.754581786 (256h14m0.625664831s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755095946 (256h14m0.626178991s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755508441 (256h14m0.626591486s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.755904064 (256h14m0.626987109s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756320638 (256h14m0.627403673s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.756732061 (256h14m0.627815106s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757153784 (256h14m0.628236819s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.757677992 (256h14m0.628761027s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.758810398 (256h14m0.629893443s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.761720710 (256h14m0.632803755s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.763846959 (256h14m0.634930004s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.765515035 (256h14m0.63659807s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.767438447 (256h14m0.638521492s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.769336853 (256h14m0.640419898s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:00.770295458 (256h14m0.641378493s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Uploading piece...
Dec 16 07:43:02.483120707 (256h14m2.354203762s) [upload=224664]: Next returned nil segment
Dec 16 07:43:02.483135836 (256h14m2.354218881s) [upload=224664]: Waiting for error group managing segments...
Dec 16 07:43:02.773246042 (256h14m2.644329087s) [noise=true , address=145.79.227.42:4122 , piece_id=N6J5XB4HBWPJYA7Y7DM3LMFNSAISNRW46U3FUOTWSEY6SL6JZ4TQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.773728249 (256h14m2.644811294s) [noise=true , address=145.79.227.7:4295 , piece_id=22NEODR6UO6BBVJAVYYFFBYJQ5YHR6BLEF6AROBCZYJV7SC6GLRA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774044531 (256h14m2.645127576s) [noise=true , address=145.79.227.49:4337 , piece_id=7DGZLFLDOQCTY4UYBW5TRJOTQ42WULNPRQ3VAJNDSXF3LGQX7DKQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.774808907 (256h14m2.645891962s) [noise=true , address=145.79.227.46:4602 , piece_id=VNYFHTZIM7V7JORHG2EWNDGYQTQMN2EL3TSHA5AC2ZEHORJS6T3A, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.777458373 (256h14m2.648541418s) [noise=true , address=145.79.227.45:4231 , piece_id=Y5MKCT2PS6YRNIVT2H4AEZBDM5RPWHPGCUP44VO2CGFDT53O3LJA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782652924 (256h14m2.653735969s) [noise=true , address=145.79.227.31:4798 , piece_id=GFW5FRG7UYOILCDWYFBVWP4FS73HBDFNC5D6K4FDVMFSWFFX24FA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.782912137 (256h14m2.653995182s) [noise=true , address=145.79.227.32:4836 , piece_id=AKF7732OSGTLRSQ65QOQR7FQI622YNIXCTVGGAZFQBOQ3IJLCLDQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.836250354 (256h14m2.707333399s) [noise=true , address=145.79.227.48:4301 , piece_id=XLX2NBE5KS3G3U23WSD7B3FHKZS6HAIXMA4BBH6BHOEQP4OTPP2Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.844189331 (256h14m2.715272366s) [noise=true , address=145.79.227.35:4482 , piece_id=CDDHADYHWUS6H67EGUMH34N2WPIDFFR57XNV7DXLQQ56BPVOD2WQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.854820926 (256h14m2.725903971s) [noise=true , address=145.79.227.50:4375 , piece_id=X6BU7ORHASMRH3M677ZHBO7XWNLORHTVWUIH2SIXSIJDSIOES44Q, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.978707800 (256h14m2.849790845s) [noise=true , address=45.91.116.13:4156 , piece_id=LPW7K6BTFQXDTWCEHGCGMIPDV2K33XYI7I7ROW3XUO7ZPKKCM4LQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:02.997550941 (256h14m2.868633986s) [noise=true , address=77.90.31.13:4629 , piece_id=IYLIU5GEU5EKVIST6P4ZO3YA2GAQVOTKBAS2B7CEJAWRRENCLESA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.017997214 (256h14m2.88908026s) [noise=true , address=45.91.116.5:4299 , piece_id=N3IURCLCCFLR5HA2DDAEQJRDBZOXFIF7HJYV6F2TEOP4KEGI2LAQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: <nil>
Dec 16 07:43:03.018025699 (256h14m2.889108754s) [seg_pos={0 0}, upload=224664]: Segment reached optimal threshold of 13 pieces.
Dec 16 07:43:03.018528225 (256h14m2.88961127s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018540770 (256h14m2.889623815s) [noise=true , address=45.91.116.4:4939 , piece_id=J37XR73CIQD54V4CR5EVHF44YZ6YV3KANKMEFJ3TDGLZ7WJP3GVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018639637 (256h14m2.889722682s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018660297 (256h14m2.889743342s) [noise=true , address=77.90.31.19:4844 , piece_id=L5VFCAYPLOGQKXXBKCB3OVGCQXT7SY3BE5WV4NHCB7II4Z7GINVA, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018744427 (256h14m2.889827492s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 16 07:43:03.018754988 (256h14m2.889838033s) [noise=true , address=45.91.116.9:4261 , piece_id=MWQEVKNMHISEUTUM5HG4SUZAOUAJ75422NV7PEEOMTEX5Z6W74EQ, seg_pos={0 0}, upload=224664]: Long tail cancellation due to optimal threshold
Dec 16 07:43:03.018779383 (256h14m2.889862428s) [seg_pos={0 0}, upload=224664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 16 07:43:03.018872130 (256h14m2.889955185s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:43:03.018879394 (256h14m2.889962439s) [upload=224664]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:43:03.316159583 (256h14m3.187242668s) [upload=224664]: Done uploading segments...
Dec 16 07:43:03.316187897 (256h14m3.187270942s) [upload=224664]: Upload finished. err: <nil>
Duration: 30.001840234s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
10.600431811s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:43:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 10.60045717s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
3.490054565s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:42:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.490087218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.809043535s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:42:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.809072532s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
15.56503946s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:42:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 15.565058907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
17.471749887s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:42:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 17.471776818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.045204138s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:41:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.045234064s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
3.469003938s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:41:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.469033755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
15.909526173s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:41:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 15.909552665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.823407355s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:41:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.823433115s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
3.44156449s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:41:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.441596491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
14.630228236s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:41:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 14.630260378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
3.581743437s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:40:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.581769699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.759733226s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:40:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.759778162s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
3.524775127s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:40:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.524804694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
23.205087717s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:40:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 23.205119326s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
3.585158459s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:39:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.585188457s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
20.332324349s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:39:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 20.332346392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
3.459988122s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:39:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.460009492s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.684465418s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:39:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.684490155s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
3.632649553s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:38:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.632669761s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.638092315s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:38:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.638117553s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
15.863031011s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:38:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 15.863060648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
3.625430155s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:38:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.625477685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.84639057s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:38:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.846415858s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
3.569867863s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:37:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.569893642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
3.630396322s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:37:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.630422601s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
20.966034391s trace:
https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48
FAIL (2025-12-16 07:37:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded
======LOGS======
h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece...
Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment
Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments...
Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil>
Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces.
Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection
Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection
Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold
Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments...
Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil>
Duration: 20.966068945s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.02036593s trace:
https://jaeger.dev.storj.io/trace/3251959fa0d27548
FAIL (2025-12-16 07:37:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.020393662s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
3.592738573s trace:
https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3
FAIL (2025-12-16 07:36:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.592764623s
core
uplink
us-select-1
us1-placement-20
|