Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
8.796632385s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:10:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 8.796686547s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
17.76972987s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:10:08 UTC)
orbiter warning: context deadline exceeded
Duration: 17.769785386s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
4.095778626s trace:
https://jaeger.dev.storj.io/trace/41b9575307efc1e3
FAIL (2025-12-05 14:10:07 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.095816539s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
8.624385618s trace:
https://jaeger.dev.storj.io/trace/70e54e58d52fa95e
FAIL (2025-12-05 14:10:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded
======LOGS======
4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment
Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments...
Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces.
Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection
Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection
Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection
Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection
Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments...
Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil>
Duration: 8.624415555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
9.035895685s trace:
https://jaeger.dev.storj.io/trace/572c91a59c970da7
FAIL (2025-12-05 14:10:03 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 9.035926353s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X..X.......X..........................] duration:
22.790369359s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:09:59 UTC)
orbiter warning: context deadline exceeded
Duration: 22.790394838s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
10.174597106s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:09:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 10.174624818s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.972250578s trace:
https://jaeger.dev.storj.io/trace/361a2ea17a62744e
FAIL (2025-12-05 14:09:40 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil>
Duration: 2.972276157s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X.] duration:
5.000569099s trace:
https://jaeger.dev.storj.io/trace/541a9c754843727b
FAIL (2025-12-05 14:09:36 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000582234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
8.194241032s trace:
https://jaeger.dev.storj.io/trace/70e54e58d52fa95e
FAIL (2025-12-05 14:09:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded
======LOGS======
4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment
Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments...
Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces.
Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection
Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection
Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection
Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection
Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments...
Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil>
Duration: 8.194259056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.912360597s trace:
https://jaeger.dev.storj.io/trace/41b9575307efc1e3
FAIL (2025-12-05 14:09:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.912392477s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
7.516924578s trace:
https://jaeger.dev.storj.io/trace/572c91a59c970da7
FAIL (2025-12-05 14:09:24 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 7.516959303s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.002572541s trace:
https://jaeger.dev.storj.io/trace/70e54e58d52fa95e
FAIL (2025-12-05 14:09:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q) context deadline exceeded; piecestore: (Node ID: 12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw, Piece ID: UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ) context deadline exceeded; piecestore: (Node ID: 12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4, Piece ID: 4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA) context deadline exceeded; piecestore: (Node ID: 12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, Piece ID: SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ) context deadline exceeded; piecestore: (Node ID: 1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, Piece ID: CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA) context deadline exceeded
======LOGS======
4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669180895 (45h13m48.140391634s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.669641381 (45h13m48.14085212s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670111447 (45h13m48.141322196s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.670601259 (45h13m48.141811999s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671075502 (45h13m48.142286241s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.671598037 (45h13m48.142808776s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672127346 (45h13m48.143338105s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.672648588 (45h13m48.143859327s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673155192 (45h13m48.144365931s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.673691684 (45h13m48.144902423s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.674805966 (45h13m48.146016705s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:11.675360772 (45h13m48.146571511s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Uploading piece...
Dec 5 14:09:13.239845356 (45h13m49.711056085s) [upload=39772]: Next returned nil segment
Dec 5 14:09:13.239860604 (45h13m49.711071334s) [upload=39772]: Waiting for error group managing segments...
Dec 5 14:09:13.643777330 (45h13m50.114988069s) [noise=true , address=77.90.31.13:4654 , piece_id=CEAQAXPEBEPYGOMBFNVRRYRCT67SMMNEVDBADWIOPQXEJCN3EGZQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.697999800 (45h13m50.169210539s) [noise=true , address=77.90.31.18:4838 , piece_id=CFMB4CDU7DWOF3L5T7DASMJB33RVAYSTMM4WFNUSUNV4H6CGPNXQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.713127946 (45h13m50.184338685s) [noise=true , address=77.90.31.15:4729 , piece_id=IY7G4GQVUEYHNO4HJAD3CHGX4SQQNGWPMUXUJJKLC7MUTUUFGFWA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.718498752 (45h13m50.189709491s) [noise=true , address=77.90.31.21:4909 , piece_id=EPHUILAEI77HCSYICLO2IMEGVISUXUEQGKNAHV52VDAGDCAP3RKQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.726522451 (45h13m50.19773318s) [noise=true , address=77.90.31.17:4794 , piece_id=ALZXJQTN64GGUOZPJMGJ6CRXQBO2R5PRVNUBQHRKWYGY7KJF6CVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.749057442 (45h13m50.220268171s) [noise=true , address=45.91.116.10:4091 , piece_id=ZS3JXULFI4AGS3HIIX2POYQ3XLBG6GPFAWBAV2NBRLMO447DH7ZA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.750808776 (45h13m50.222019515s) [noise=true , address=77.90.31.10:4542 , piece_id=XDD4QZAKCWJK5A3F5FDL4NRNPEQZF3GKODCGVT2V47AH6KVEOOWQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.754762756 (45h13m50.225973485s) [noise=true , address=77.90.31.11:4556 , piece_id=WBCLEOB7XBAT2MPT6M7NLLNQM2JR3S22BV6UZBSGRDFIJP5JS2LA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.759242185 (45h13m50.230452925s) [noise=true , address=77.90.31.12:4616 , piece_id=SUX7UBSIQIIYU327MGSXKLJBPXFW6UEVI3NJRAWVEZ3GBUN2QCJQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.774028982 (45h13m50.245239711s) [noise=true , address=77.90.31.19:4856 , piece_id=MQGKTFSYGQSOMNUEIVOGGS5ZEVM52CLW4TEG7DA5ICOK73GEAI5Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.786684680 (45h13m50.257895419s) [noise=true , address=45.91.116.9:4272 , piece_id=4W72IA43JPPLQ5ZDZHZDW6AFEZDVG7AHJT7555P4BY2Z3QYMLKNA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.790669208 (45h13m50.261879947s) [noise=true , address=45.91.116.11:4147 , piece_id=UALY4LTNPIS62L46OIH4PTKRES72I7LA55UL7B2UCZLRT7DQ7RVQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.850747877 (45h13m50.321958617s) [noise=true , address=45.91.116.8:4071 , piece_id=3YI67CEXFW75VRROYLSABR55DNVFHE4SR6EUP57CPRIIB63GZOVA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884817071 (45h13m50.3560278s) [noise=true , address=45.91.116.5:4329 , piece_id=MBHG2UWXQMEZPJJHW67KJMJTIVT4QWPXKDIRBRVTBAFN46J3M42Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: <nil>
Dec 5 14:09:13.884838361 (45h13m50.35604909s) [seg_pos={0 0}, upload=39772]: Segment reached optimal threshold of 14 pieces.
Dec 5 14:09:13.885054584 (45h13m50.356265313s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JNee734A635XWMM3Q3N6siDsvKY3FEMQerDjYZrQR6orGBaY3): protocol: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59612->145.79.227.15:4907: use of closed network connection
Dec 5 14:09:13.885064302 (45h13m50.356275041s) [noise=true , address=145.79.227.15:4907 , piece_id=BXEPFHK6AENJZM2APP5LPV2LHJ4L7CBFQF5JSQJQBC6F5AKDGYQA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885416693 (45h13m50.356627432s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3): protocol: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37344->145.79.227.19:4097: use of closed network connection
Dec 5 14:09:13.885427744 (45h13m50.356638483s) [noise=true , address=145.79.227.19:4097 , piece_id=MKAKS25RPIEVHT2WJJ7QYY4FBS7JCIM3RKY7MWRJ7FZTZ2YZXMGQ, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885527824 (45h13m50.356738563s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43992->145.79.227.32:4855: use of closed network connection
Dec 5 14:09:13.885535649 (45h13m50.356746388s) [noise=true , address=145.79.227.32:4855 , piece_id=N3F2LVAV3UNBWZ7V7HFLNVU5NGWD6X2RB6WKVEVGBJMHUUIJP3PA, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885614951 (45h13m50.35682569s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44254->64.49.13.169:4705: use of closed network connection
Dec 5 14:09:13.885620010 (45h13m50.356830749s) [noise=true , address=64.49.13.169:4705 , piece_id=TTCHF4ENHBWLOJQCAJKMG6BUZGHDDEU3KW5AU5Q3CFCOTKIONE4Q, seg_pos={0 0}, upload=39772]: Long tail cancellation due to optimal threshold
Dec 5 14:09:13.885636241 (45h13m50.35684697s) [seg_pos={0 0}, upload=39772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 5 14:09:13.885731973 (45h13m50.356942722s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 14:09:13.885739618 (45h13m50.356950357s) [upload=39772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 14:09:14.639752295 (45h13m51.110963024s) [upload=39772]: Done uploading segments...
Dec 5 14:09:14.639788193 (45h13m51.110998932s) [upload=39772]: Upload finished. err: <nil>
Duration: 1m0.002605253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
25.524179606s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:09:12 UTC)
orbiter warning: context deadline exceeded
Duration: 25.524208182s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
18.458468s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:09:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 18.458511142s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001258713s trace:
https://jaeger.dev.storj.io/trace/41b9575307efc1e3
FAIL (2025-12-05 14:09:09 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001279242s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
24.148112663s trace:
https://jaeger.dev.storj.io/trace/572c91a59c970da7
FAIL (2025-12-05 14:09:07 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 24.148199659s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
18.756580349s trace:
https://jaeger.dev.storj.io/trace/361a2ea17a62744e
FAIL (2025-12-05 14:08:59 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil>
Duration: 18.756703973s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.X] duration:
5.002398432s trace:
https://jaeger.dev.storj.io/trace/541a9c754843727b
FAIL (2025-12-05 14:08:36 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002421235s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
24.348131152s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:08:36 UTC)
orbiter warning: context deadline exceeded
Duration: 24.348157943s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X..X.......X.........................] duration:
12.420785087s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:08:35 UTC)
orbiter warning: context deadline exceeded
Duration: 12.420813181s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001159232s trace:
https://jaeger.dev.storj.io/trace/361a2ea17a62744e
FAIL (2025-12-05 14:08:29 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 5 14:08:02.649750072 (45h12m39.120960811s) [upload=39762]: Starting upload Dec 5 14:08:02.650012922 (45h12m39.121223681s) [upload=39762]: Done starting upload Dec 5 14:08:02.650140385 (45h12m39.121351124s) [upload=39762]: Uploading segments... Dec 5 14:08:02.650204036 (45h12m39.121414775s) [upload=39762]: Got next segment. Inline: true Dec 5 14:08:02.650210659 (45h12m39.121421398s) [upload=39762]: Waiting for error group managing segments... Dec 5 14:08:02.650285611 (45h12m39.12149635s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 5 14:08:02.650292053 (45h12m39.121502792s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 5 14:08:02.650295811 (45h12m39.12150655s) [upload=39762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 5 14:08:05.223576463 (45h12m41.694787212s) [upload=39762]: Done uploading segments... Dec 5 14:08:05.223604205 (45h12m41.694814934s) [upload=39762]: Upload finished. err: <nil>
Duration: 30.001185502s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000826348s trace:
https://jaeger.dev.storj.io/trace/572c91a59c970da7
FAIL (2025-12-05 14:08:28 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000860663s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX.] duration:
5.000987817s trace:
https://jaeger.dev.storj.io/trace/568fcac588f079d0
FAIL (2025-12-05 14:08:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001013346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X..X.......X........................] duration:
14.60404412s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:08:16 UTC)
orbiter warning: context deadline exceeded
Duration: 14.604074368s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXXX] duration:
5.001653274s trace:
https://jaeger.dev.storj.io/trace/568fcac588f079d0
FAIL (2025-12-05 14:07:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001676067s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
10.894773523s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:07:54 UTC)
orbiter warning: context deadline exceeded
Duration: 10.89479857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
13.664853954s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:07:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 13.664883529s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X..X.......X.......................] duration:
15.900578018s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:07:35 UTC)
orbiter warning: context deadline exceeded
Duration: 15.900610179s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
12.820511941s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:07:35 UTC)
orbiter warning: context deadline exceeded
Duration: 12.8205265s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
10.164628923s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:07:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 10.164673418s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
24.606599326s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:07:06 UTC)
orbiter warning: context deadline exceeded
Duration: 24.606628542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXXX] duration:
5.002181149s trace:
https://jaeger.dev.storj.io/trace/54c1898517bee92e
FAIL (2025-12-05 14:07:01 UTC)
orbiter warning: context deadline exceeded
Duration: 5.002207569s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X..X.......X......................] duration:
19.17816033s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:06:29 UTC)
orbiter warning: context deadline exceeded
Duration: 19.178189035s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
8.829468872s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:06:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 8.829499862s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
17.067091561s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:06:12 UTC)
orbiter warning: context deadline exceeded
Duration: 17.067158346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXXX] duration:
5.001824229s trace:
https://jaeger.dev.storj.io/trace/4753cbc31b32fb63
FAIL (2025-12-05 14:05:55 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001852144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X..X.......X.....................] duration:
18.237874819s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:05:44 UTC)
orbiter warning: context deadline exceeded
Duration: 18.237905678s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XXX] duration:
5.001636952s trace:
https://jaeger.dev.storj.io/trace/120e9c3c5d76850c
FAIL (2025-12-05 14:05:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001665857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
9.493294119s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:05:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 9.493330157s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X..X.......X....................] duration:
20.543716768s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:05:09 UTC)
orbiter warning: context deadline exceeded
Duration: 20.543808634s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.831555234s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:04:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.831580972s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
10.626742622s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:04:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 10.626765436s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [X..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..XX] duration:
5.001601562s trace:
https://jaeger.dev.storj.io/trace/84c387adf9dcffa
FAIL (2025-12-05 14:04:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001626208s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X..X.......X...................] duration:
13.971790368s trace:
https://jaeger.dev.storj.io/trace/5cac00e234ac2601
FAIL (2025-12-05 14:04:18 UTC)
orbiter warning: context deadline exceeded
Duration: 13.971827399s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
13.842052128s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 14:04:14 UTC)
orbiter warning: context deadline exceeded
Duration: 13.842083999s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..X] duration:
5.001355006s trace:
https://jaeger.dev.storj.io/trace/31568fcff4d1ee26
FAIL (2025-12-05 14:04:07 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001380585s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
9.496262308s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:03:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 9.496292696s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXXXXXX.X..XXXXX..] duration:
5.001281194s trace:
https://jaeger.dev.storj.io/trace/415a710154adadf2
FAIL (2025-12-05 14:03:29 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001312554s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
9.865487032s trace:
https://jaeger.dev.storj.io/trace/65df260343fa2062
FAIL (2025-12-05 14:03:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: 2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ) context deadline exceeded; piecestore: (Node ID: 1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, Piece ID: UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA) context deadline exceeded
======LOGS======
Dec 5 13:50:13.914958542 (44h54m50.386169281s) [upload=39497]: Starting upload
Dec 5 13:50:13.915042042 (44h54m50.386252771s) [upload=39497]: Done starting upload
Dec 5 13:50:13.915237493 (44h54m50.386448232s) [upload=39497]: Uploading segments...
Dec 5 13:50:13.915339918 (44h54m50.386550688s) [upload=39497]: Got next segment. Inline: false
Dec 5 13:50:13.915395575 (44h54m50.386606314s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 5 13:50:13.915400735 (44h54m50.386611464s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 5 13:50:14.177967782 (44h54m50.649178531s) [seg_pos={0 0}, upload=39497]: Begin upload segment...
Dec 5 13:50:14.178049498 (44h54m50.649260237s) [seg_pos={0 0}, upload=39497]: Done begin upload segment.
Dec 5 13:50:14.178089053 (44h54m50.649299792s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178100966 (44h54m50.649311715s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178155129 (44h54m50.649365868s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.178737678 (44h54m50.649948417s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179277977 (44h54m50.650488716s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.179826452 (44h54m50.651037191s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.180516445 (44h54m50.651727174s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.181184327 (44h54m50.652395066s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.182617817 (44h54m50.653828556s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.183276120 (44h54m50.654486859s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.184534556 (44h54m50.655745295s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.186480883 (44h54m50.657691622s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.188448450 (44h54m50.659659189s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.190696029 (44h54m50.661906768s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.193175661 (44h54m50.6643864s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:14.195649231 (44h54m50.66685997s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Uploading piece...
Dec 5 13:50:15.979218426 (44h54m52.450429165s) [upload=39497]: Next returned nil segment
Dec 5 13:50:15.979231121 (44h54m52.45044185s) [upload=39497]: Waiting for error group managing segments...
Dec 5 13:50:16.323526524 (44h54m52.794737263s) [noise=true , address=145.79.227.21:4475 , piece_id=5IPJJ5RHSXCJPUUXXD62EKMNSWSHPIVEWWYS4X77GE7OT6NCTJUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.347685978 (44h54m52.818896717s) [noise=true , address=145.79.227.23:4546 , piece_id=UNWONHZ6EIRFROZ5L5YAEFBSMFZHEN4VKAKYM2DWTW4BGCTNMAMA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351351307 (44h54m52.822562036s) [noise=true , address=145.79.227.10:4402 , piece_id=IHNZYV6LJREQ63GBFJY6BVB54QLSVCZIIUUTLOG7YOOOXNGHQ7WA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.351375874 (44h54m52.822586603s) [noise=true , address=145.79.227.38:4978 , piece_id=N7OVLKMMMZW75CCZDX5QYOG2SDCO7AKO2FITHFDO3EMDSPJJPAYA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.359501778 (44h54m52.830712517s) [noise=true , address=145.79.227.31:4799 , piece_id=2C63VGDP7JVQVWSLYLHZEHC4SB7PU6NFUZWLCTEO7AGDP32X3JOQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.360729425 (44h54m52.831940164s) [noise=true , address=145.79.227.4:4187 , piece_id=VMP5DCNJDH5EAO6LOHSOCP4GGOYI6EV3YHCW5GXB2BRTOO67YQMQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.371054476 (44h54m52.842265225s) [noise=true , address=145.79.227.41:4087 , piece_id=2KY6XA6532G23SRYSNBYNYZ2Y65ZLMXDC2ERGQ6PH4UJQZDUZRUA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.392350999 (44h54m52.863561738s) [noise=true , address=77.90.31.14:4663 , piece_id=7WW2DVMZSNJTG3J4NANUAIZKMLICAYLP5RAOKKWCTCUTJAISG7SQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.437887268 (44h54m52.909098007s) [noise=true , address=145.79.227.21:4474 , piece_id=J5V4QU72P7D2FGZ7NWO5UT3E7S25CTN4MCSPQMSOWK7R6GK4B4HA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.642787347 (44h54m53.113998086s) [noise=true , address=145.79.227.43:4157 , piece_id=PFPPMUR22PUIHIPCNXQGN5CYFM4K4OG2EMC5LAMUH26RZKETHZAQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:16.987804677 (44h54m53.459015416s) [noise=true , address=145.79.227.25:4619 , piece_id=RV4CB7VT7T3YVFFBGXA6EVYMGEOWBU3C3XYUTEMRZDBVHRNVEVLA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.014774723 (44h54m53.485985462s) [noise=true , address=145.79.227.14:4838 , piece_id=6EVD7RSXQJOFT645D7ILWLUWUBYFP6IRR5GESYOAWXF2IXKJEUCA, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413923466 (44h54m53.885134205s) [noise=true , address=145.79.227.16:4909 , piece_id=JYAT6OFRJQNEZPORFRHUCTFQDPVM2ZTXEUBWGNY6BLADCTZIS66Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: <nil>
Dec 5 13:50:17.413981627 (44h54m53.885192367s) [seg_pos={0 0}, upload=39497]: Segment reached optimal threshold of 13 pieces.
Dec 5 13:50:17.414271349 (44h54m53.885482088s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56230->145.79.227.39:4015: use of closed network connection
Dec 5 13:50:17.414282390 (44h54m53.885493119s) [noise=true , address=145.79.227.39:4015 , piece_id=V6WH4XCY2YVP2B3DR4HLI7IFH7MGGPAIBAZHHDBHQQ4NUTCNHP7Q, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414355008 (44h54m53.885565747s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414365318 (44h54m53.885576057s) [noise=true , address=145.79.227.34:4906 , piece_id=L2SWU22TLNZNS3LO2E3HYCKOVPO6CIOZWOACPQV7CGMURN2HEQRQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414811999 (44h54m53.886022758s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF
Dec 5 13:50:17.414824472 (44h54m53.886035211s) [noise=true , address=145.79.227.20:4438 , piece_id=P64J3PKA4V3SQWAC36KWBVXDBE4KXYJHUXWYHNFZHCQEGQZUWHFQ, seg_pos={0 0}, upload=39497]: Long tail cancellation due to optimal threshold
Dec 5 13:50:17.414850933 (44h54m53.886061672s) [seg_pos={0 0}, upload=39497]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 5 13:50:17.414965771 (44h54m53.88617652s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 5 13:50:17.414973846 (44h54m53.886184585s) [upload=39497]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 5 13:50:17.720870703 (44h54m54.192081452s) [upload=39497]: Done uploading segments...
Dec 5 13:50:17.720894969 (44h54m54.192105708s) [upload=39497]: Upload finished. err: <nil>
Duration: 9.865519243s
core
uplink
us-select-1
|