Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [......X..............X.........X.........X............X...XX] duration:
5.001247471s trace:
https://jaeger.dev.storj.io/trace/7bb354c3290fb371
FAIL (2025-12-31 05:16:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001259794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.381441129s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:16:24 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.381510491s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X.............................X.............] duration:
27.695604704s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:16:13 UTC)
orbiter warning: context deadline exceeded
Duration: 27.695623351s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X.......X.........XX...X.X....X......X.XXX...XX.X.......X.] duration:
5.943452563s trace:
https://jaeger.dev.storj.io/trace/5bd041c1e3cd6569
FAIL (2025-12-31 05:16:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 05:15:35.066844761 (613h46m34.937927806s) [upload=537871]: Starting upload
Dec 31 05:15:35.066909465 (613h46m34.93799251s) [upload=537871]: Done starting upload
Dec 31 05:15:35.067024854 (613h46m34.938107899s) [upload=537871]: Uploading segments...
Dec 31 05:15:35.067141366 (613h46m34.938224401s) [upload=537871]: Got next segment. Inline: false
Dec 31 05:15:35.067182435 (613h46m34.93826548s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:15:35.067188676 (613h46m34.938271721s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:15:35.322066185 (613h46m35.19314923s) [seg_pos={0 0}, upload=537871]: Begin upload segment...
Dec 31 05:15:35.322118595 (613h46m35.19320164s) [seg_pos={0 0}, upload=537871]: Done begin upload segment.
Dec 31 05:15:35.322151026 (613h46m35.193234061s) [noise=true , address=145.79.227.12:4765 , piece_id=I7YYMZHOJR4ZOV6GTMDFSFAULO4N73ICYCBUY545SODW6LUAKWVA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.322188588 (613h46m35.193271623s) [noise=true , address=45.91.116.6:4973 , piece_id=5AYA3XYDC73AGNRXUASIEHIUZ6WTNNIJD4X36OY5RWVUYELW7HJQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.322793790 (613h46m35.193876825s) [noise=true , address=145.79.227.16:4912 , piece_id=V2M7NKY4552CTWD7PLFULSUDIP36THPULL66HKYH4WHZ4ANXFP6A, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323213379 (613h46m35.194296414s) [noise=true , address=45.91.116.12:4226 , piece_id=42AFSCHTACDDRDRW263KXYHZBBIX7FEDHA3DQ4VFXNCXSOPHCTHA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323381830 (613h46m35.194464875s) [noise=true , address=145.79.227.23:4545 , piece_id=IOIRJ6DZ6VGCDOHPIDJOI6PEIKTEBAKFNSN44EHA33WQXGYZWSGQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323768065 (613h46m35.19485111s) [noise=true , address=145.79.227.14:4839 , piece_id=T74GQ3GTMI23MM5FCYSBH3OMMVCAGFEA3HLMENXO2I3ZHBTYXZBQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323946476 (613h46m35.195029511s) [noise=true , address=145.79.227.37:4567 , piece_id=MUNCMLWG2PK2GXCDVTPLFPHSXHZWWNILNVTC7RKJ43GWTJZT24ZA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324136587 (613h46m35.195219632s) [noise=true , address=145.79.227.52:4445 , piece_id=5STPD3QSUV7HN7KIFH2SM5LHMCYC6ZDJ3F3URVVNUUVVY7TZGUSA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324313865 (613h46m35.19539691s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324497735 (613h46m35.19558078s) [noise=true , address=145.79.227.27:4690 , piece_id=665SXHU3XQ3DN7QBFZGLXK7UJ6L5A2YVCJJKVWBWL6QSMFVXRBLA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324658311 (613h46m35.195741356s) [noise=true , address=45.91.116.13:4155 , piece_id=EAUWJRBDLPGDTIMUPX3ZN3A3JLVCA6I3XTX4R7VWIRE4GZSAMAKA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324870304 (613h46m35.195953349s) [noise=true , address=145.79.227.19:4079 , piece_id=HST6ZX6L74Q5PE47QPJRUYUS22PK3WII3WML3FP3UEU3EROLDSRQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325024247 (613h46m35.196107292s) [noise=true , address=145.79.227.25:4617 , piece_id=WSYCPUS3ARTVPPD5ONWDFUYQRVFW424SU4CRYKO77H3WPF2VEIIA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325263643 (613h46m35.196346688s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325401465 (613h46m35.196484511s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325782040 (613h46m35.196865085s) [noise=true , address=77.90.31.13:4627 , piece_id=DQQCCGIGYYGFLXFXABOQQPJM3P6YII4TTPLQYLPMVQ7I62ZERXDA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:37.026140327 (613h46m36.897223362s) [upload=537871]: Next returned nil segment
Dec 31 05:15:37.026166618 (613h46m36.897249653s) [upload=537871]: Waiting for error group managing segments...
Dec 31 05:15:37.305959378 (613h46m37.177042423s) [noise=true , address=145.79.227.14:4839 , piece_id=T74GQ3GTMI23MM5FCYSBH3OMMVCAGFEA3HLMENXO2I3ZHBTYXZBQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.310359046 (613h46m37.181442091s) [noise=true , address=145.79.227.23:4545 , piece_id=IOIRJ6DZ6VGCDOHPIDJOI6PEIKTEBAKFNSN44EHA33WQXGYZWSGQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.312483941 (613h46m37.183566986s) [noise=true , address=145.79.227.25:4617 , piece_id=WSYCPUS3ARTVPPD5ONWDFUYQRVFW424SU4CRYKO77H3WPF2VEIIA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.322924652 (613h46m37.194007687s) [noise=true , address=145.79.227.16:4912 , piece_id=V2M7NKY4552CTWD7PLFULSUDIP36THPULL66HKYH4WHZ4ANXFP6A, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.331935590 (613h46m37.203018635s) [noise=true , address=145.79.227.12:4765 , piece_id=I7YYMZHOJR4ZOV6GTMDFSFAULO4N73ICYCBUY545SODW6LUAKWVA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.332337055 (613h46m37.2034201s) [noise=true , address=145.79.227.37:4567 , piece_id=MUNCMLWG2PK2GXCDVTPLFPHSXHZWWNILNVTC7RKJ43GWTJZT24ZA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.353681758 (613h46m37.224764803s) [noise=true , address=145.79.227.27:4690 , piece_id=665SXHU3XQ3DN7QBFZGLXK7UJ6L5A2YVCJJKVWBWL6QSMFVXRBLA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.397755458 (613h46m37.268838503s) [noise=true , address=145.79.227.19:4079 , piece_id=HST6ZX6L74Q5PE47QPJRUYUS22PK3WII3WML3FP3UEU3EROLDSRQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.397912388 (613h46m37.268995433s) [noise=true , address=145.79.227.52:4445 , piece_id=5STPD3QSUV7HN7KIFH2SM5LHMCYC6ZDJ3F3URVVNUUVVY7TZGUSA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.604700658 (613h46m37.475783693s) [noise=true , address=77.90.31.13:4627 , piece_id=DQQCCGIGYYGFLXFXABOQQPJM3P6YII4TTPLQYLPMVQ7I62ZERXDA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.726276996 (613h46m37.597360041s) [noise=true , address=45.91.116.13:4155 , piece_id=EAUWJRBDLPGDTIMUPX3ZN3A3JLVCA6I3XTX4R7VWIRE4GZSAMAKA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.963272815 (613h46m37.83435585s) [noise=true , address=45.91.116.12:4226 , piece_id=42AFSCHTACDDRDRW263KXYHZBBIX7FEDHA3DQ4VFXNCXSOPHCTHA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:38.089252616 (613h46m37.960335661s) [noise=true , address=45.91.116.6:4973 , piece_id=5AYA3XYDC73AGNRXUASIEHIUZ6WTNNIJD4X36OY5RWVUYELW7HJQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:38.089272334 (613h46m37.960355369s) [seg_pos={0 0}, upload=537871]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:15:38.089467044 (613h46m37.960550089s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089472976 (613h46m37.960556011s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089585350 (613h46m37.960668405s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089596651 (613h46m37.960679696s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089669830 (613h46m37.960752875s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089678366 (613h46m37.960761401s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089698215 (613h46m37.96078125s) [seg_pos={0 0}, upload=537871]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:15:38.089777034 (613h46m37.960860069s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:15:38.089782545 (613h46m37.96086559s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:15:38.399351874 (613h46m38.270434909s) [upload=537871]: Done uploading segments...
Dec 31 05:15:38.399372524 (613h46m38.270455559s) [upload=537871]: Upload finished. err: <nil>
Duration: 5.943483082s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration:
35.202867867s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:16:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 35.20289573s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...X.......X.........XX...X.X....X......X.XXX...XX.X.......X] duration:
30.005259349s trace:
https://jaeger.dev.storj.io/trace/5bd041c1e3cd6569
FAIL (2025-12-31 05:16:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 05:15:35.066844761 (613h46m34.937927806s) [upload=537871]: Starting upload
Dec 31 05:15:35.066909465 (613h46m34.93799251s) [upload=537871]: Done starting upload
Dec 31 05:15:35.067024854 (613h46m34.938107899s) [upload=537871]: Uploading segments...
Dec 31 05:15:35.067141366 (613h46m34.938224401s) [upload=537871]: Got next segment. Inline: false
Dec 31 05:15:35.067182435 (613h46m34.93826548s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:15:35.067188676 (613h46m34.938271721s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:15:35.322066185 (613h46m35.19314923s) [seg_pos={0 0}, upload=537871]: Begin upload segment...
Dec 31 05:15:35.322118595 (613h46m35.19320164s) [seg_pos={0 0}, upload=537871]: Done begin upload segment.
Dec 31 05:15:35.322151026 (613h46m35.193234061s) [noise=true , address=145.79.227.12:4765 , piece_id=I7YYMZHOJR4ZOV6GTMDFSFAULO4N73ICYCBUY545SODW6LUAKWVA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.322188588 (613h46m35.193271623s) [noise=true , address=45.91.116.6:4973 , piece_id=5AYA3XYDC73AGNRXUASIEHIUZ6WTNNIJD4X36OY5RWVUYELW7HJQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.322793790 (613h46m35.193876825s) [noise=true , address=145.79.227.16:4912 , piece_id=V2M7NKY4552CTWD7PLFULSUDIP36THPULL66HKYH4WHZ4ANXFP6A, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323213379 (613h46m35.194296414s) [noise=true , address=45.91.116.12:4226 , piece_id=42AFSCHTACDDRDRW263KXYHZBBIX7FEDHA3DQ4VFXNCXSOPHCTHA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323381830 (613h46m35.194464875s) [noise=true , address=145.79.227.23:4545 , piece_id=IOIRJ6DZ6VGCDOHPIDJOI6PEIKTEBAKFNSN44EHA33WQXGYZWSGQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323768065 (613h46m35.19485111s) [noise=true , address=145.79.227.14:4839 , piece_id=T74GQ3GTMI23MM5FCYSBH3OMMVCAGFEA3HLMENXO2I3ZHBTYXZBQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.323946476 (613h46m35.195029511s) [noise=true , address=145.79.227.37:4567 , piece_id=MUNCMLWG2PK2GXCDVTPLFPHSXHZWWNILNVTC7RKJ43GWTJZT24ZA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324136587 (613h46m35.195219632s) [noise=true , address=145.79.227.52:4445 , piece_id=5STPD3QSUV7HN7KIFH2SM5LHMCYC6ZDJ3F3URVVNUUVVY7TZGUSA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324313865 (613h46m35.19539691s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324497735 (613h46m35.19558078s) [noise=true , address=145.79.227.27:4690 , piece_id=665SXHU3XQ3DN7QBFZGLXK7UJ6L5A2YVCJJKVWBWL6QSMFVXRBLA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324658311 (613h46m35.195741356s) [noise=true , address=45.91.116.13:4155 , piece_id=EAUWJRBDLPGDTIMUPX3ZN3A3JLVCA6I3XTX4R7VWIRE4GZSAMAKA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.324870304 (613h46m35.195953349s) [noise=true , address=145.79.227.19:4079 , piece_id=HST6ZX6L74Q5PE47QPJRUYUS22PK3WII3WML3FP3UEU3EROLDSRQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325024247 (613h46m35.196107292s) [noise=true , address=145.79.227.25:4617 , piece_id=WSYCPUS3ARTVPPD5ONWDFUYQRVFW424SU4CRYKO77H3WPF2VEIIA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325263643 (613h46m35.196346688s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325401465 (613h46m35.196484511s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:35.325782040 (613h46m35.196865085s) [noise=true , address=77.90.31.13:4627 , piece_id=DQQCCGIGYYGFLXFXABOQQPJM3P6YII4TTPLQYLPMVQ7I62ZERXDA, seg_pos={0 0}, upload=537871]: Uploading piece...
Dec 31 05:15:37.026140327 (613h46m36.897223362s) [upload=537871]: Next returned nil segment
Dec 31 05:15:37.026166618 (613h46m36.897249653s) [upload=537871]: Waiting for error group managing segments...
Dec 31 05:15:37.305959378 (613h46m37.177042423s) [noise=true , address=145.79.227.14:4839 , piece_id=T74GQ3GTMI23MM5FCYSBH3OMMVCAGFEA3HLMENXO2I3ZHBTYXZBQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.310359046 (613h46m37.181442091s) [noise=true , address=145.79.227.23:4545 , piece_id=IOIRJ6DZ6VGCDOHPIDJOI6PEIKTEBAKFNSN44EHA33WQXGYZWSGQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.312483941 (613h46m37.183566986s) [noise=true , address=145.79.227.25:4617 , piece_id=WSYCPUS3ARTVPPD5ONWDFUYQRVFW424SU4CRYKO77H3WPF2VEIIA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.322924652 (613h46m37.194007687s) [noise=true , address=145.79.227.16:4912 , piece_id=V2M7NKY4552CTWD7PLFULSUDIP36THPULL66HKYH4WHZ4ANXFP6A, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.331935590 (613h46m37.203018635s) [noise=true , address=145.79.227.12:4765 , piece_id=I7YYMZHOJR4ZOV6GTMDFSFAULO4N73ICYCBUY545SODW6LUAKWVA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.332337055 (613h46m37.2034201s) [noise=true , address=145.79.227.37:4567 , piece_id=MUNCMLWG2PK2GXCDVTPLFPHSXHZWWNILNVTC7RKJ43GWTJZT24ZA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.353681758 (613h46m37.224764803s) [noise=true , address=145.79.227.27:4690 , piece_id=665SXHU3XQ3DN7QBFZGLXK7UJ6L5A2YVCJJKVWBWL6QSMFVXRBLA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.397755458 (613h46m37.268838503s) [noise=true , address=145.79.227.19:4079 , piece_id=HST6ZX6L74Q5PE47QPJRUYUS22PK3WII3WML3FP3UEU3EROLDSRQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.397912388 (613h46m37.268995433s) [noise=true , address=145.79.227.52:4445 , piece_id=5STPD3QSUV7HN7KIFH2SM5LHMCYC6ZDJ3F3URVVNUUVVY7TZGUSA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.604700658 (613h46m37.475783693s) [noise=true , address=77.90.31.13:4627 , piece_id=DQQCCGIGYYGFLXFXABOQQPJM3P6YII4TTPLQYLPMVQ7I62ZERXDA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.726276996 (613h46m37.597360041s) [noise=true , address=45.91.116.13:4155 , piece_id=EAUWJRBDLPGDTIMUPX3ZN3A3JLVCA6I3XTX4R7VWIRE4GZSAMAKA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:37.963272815 (613h46m37.83435585s) [noise=true , address=45.91.116.12:4226 , piece_id=42AFSCHTACDDRDRW263KXYHZBBIX7FEDHA3DQ4VFXNCXSOPHCTHA, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:38.089252616 (613h46m37.960335661s) [noise=true , address=45.91.116.6:4973 , piece_id=5AYA3XYDC73AGNRXUASIEHIUZ6WTNNIJD4X36OY5RWVUYELW7HJQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: <nil>
Dec 31 05:15:38.089272334 (613h46m37.960355369s) [seg_pos={0 0}, upload=537871]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:15:38.089467044 (613h46m37.960550089s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089472976 (613h46m37.960556011s) [noise=true , address=145.79.227.49:4338 , piece_id=P7QINUIFYIX3NJJ2VSOP7FJXPCTO6HEVJNOSES2XCD457K76NR5A, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089585350 (613h46m37.960668405s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089596651 (613h46m37.960679696s) [noise=true , address=77.90.31.16:4735 , piece_id=PDUXDYEPQUC6QXTKTD6ARLIGUFA7LW7MW37HIPKK6TTMSRXOZAQQ, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089669830 (613h46m37.960752875s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 31 05:15:38.089678366 (613h46m37.960761401s) [noise=true , address=77.90.31.10:4518 , piece_id=FGSFVXKXFCFUEKMSFI2KQXDX3LFCM4ASMVEKW7SZ6SM4D5PHS3AQ, seg_pos={0 0}, upload=537871]: Long tail cancellation due to optimal threshold
Dec 31 05:15:38.089698215 (613h46m37.96078125s) [seg_pos={0 0}, upload=537871]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:15:38.089777034 (613h46m37.960860069s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:15:38.089782545 (613h46m37.96086559s) [upload=537871]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:15:38.399351874 (613h46m38.270434909s) [upload=537871]: Done uploading segments...
Dec 31 05:15:38.399372524 (613h46m38.270455559s) [upload=537871]: Upload finished. err: <nil>
Duration: 30.005282604s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.866585756s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:15:49 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.866616555s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.32293184s trace:
https://jaeger.dev.storj.io/trace/568f439f0c3260e7
FAIL (2025-12-31 05:15:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.322954442s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.......X..............X.........X.........X............X...X] duration:
5.00052588s trace:
https://jaeger.dev.storj.io/trace/395f39ab704c6e87
FAIL (2025-12-31 05:15:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000551618s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
9.601036754s trace:
https://jaeger.dev.storj.io/trace/568f439f0c3260e7
FAIL (2025-12-31 05:15:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 9.601064047s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X...................................................X....] duration:
47.595228707s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:15:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 47.595256549s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X.............................X............] duration:
20.459271759s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:15:25 UTC)
orbiter warning: context deadline exceeded
Duration: 20.45951902s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
10.653697135s trace:
https://jaeger.dev.storj.io/trace/568f439f0c3260e7
FAIL (2025-12-31 05:15:17 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.653744235s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.270025421s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:15:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.270058603s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X.......X.........XX...X.X....X......X.XXX...XX.X.......] duration:
6.821491986s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:14:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 6.821515289s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [........X..............X.........X.........X............X...] duration:
3.424893035s trace:
https://jaeger.dev.storj.io/trace/f4ca10848c1d294
FAIL (2025-12-31 05:14:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.42491116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X.............................X...........] duration:
6.291458823s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:14:36 UTC)
orbiter warning: context deadline exceeded
Duration: 6.291503649s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X...................................................X...] duration:
15.233890389s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:14:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 15.233926198s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........X..............X.........X.........X............X..] duration:
3.44084098s trace:
https://jaeger.dev.storj.io/trace/f4ca10848c1d294
FAIL (2025-12-31 05:14:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.440919891s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.226277066s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:14:13 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.226309288s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X.............................X..........] duration:
6.210822163s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:14:06 UTC)
orbiter warning: context deadline exceeded
Duration: 6.210884581s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration:
24.878034053s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:13:51 UTC)
orbiter warning: context deadline exceeded
Duration: 24.87805384s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.328898697s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:13:49 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.328943613s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X.......X.........XX...X.X....X......X.XXX...XX.X......] duration:
6.03872581s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:13:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 6.038753232s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........X..............X.........X.........X............X.] duration:
3.599885425s trace:
https://jaeger.dev.storj.io/trace/f4ca10848c1d294
FAIL (2025-12-31 05:13:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.599909511s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration:
19.103948369s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:13:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 19.103972183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.020654721s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:13:16 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.020686801s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........X..............X.........X.........X............X] duration:
5.001591721s trace:
https://jaeger.dev.storj.io/trace/f4ca10848c1d294
FAIL (2025-12-31 05:13:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001633682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration:
6.183727327s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:12:51 UTC)
orbiter warning: context deadline exceeded
Duration: 6.183757183s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X...........X..............X.........X.........X............] duration:
3.591056936s trace:
https://jaeger.dev.storj.io/trace/2b1c6f2c257d6b6b
FAIL (2025-12-31 05:12:45 UTC)
orbiter warning: context deadline exceeded
Duration: 3.591089957s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X...................................................X.] duration:
7.449822585s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:12:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 7.449841701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X...................................................X] duration:
1m0.005247648s trace:
https://jaeger.dev.storj.io/trace/3c8cfeb4fd604277
FAIL (2025-12-31 05:12:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
0:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571801359 (613h42m33.442884404s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.571823482 (613h42m33.442906517s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572161024 (613h42m33.443244059s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572259872 (613h42m33.443342907s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572543683 (613h42m33.443626728s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572623084 (613h42m33.44370613s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572957431 (613h42m33.444040466s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.572981246 (613h42m33.444064291s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573306215 (613h42m33.44438925s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.573674817 (613h42m33.444757852s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574090038 (613h42m33.445173073s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.574471895 (613h42m33.44555493s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.575750298 (613h42m33.446833343s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:33.576111316 (613h42m33.447194362s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Uploading piece...
Dec 31 05:11:34.968524338 (613h42m34.839607383s) [upload=537816]: Next returned nil segment
Dec 31 05:11:34.968538835 (613h42m34.83962188s) [upload=537816]: Waiting for error group managing segments...
Dec 31 05:11:35.444651244 (613h42m35.315734279s) [noise=true , address=145.79.227.19:4082 , piece_id=VG2KPTEWPTFUZTJ6F6ZDRWJLQXAZI445CXW3YMO4MQXB5J2JMS3Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.445226108 (613h42m35.316309153s) [noise=true , address=145.79.227.46:4621 , piece_id=6COITJDWR7AGAPJYASJDX2R664SGOHWRLZIG7GMAHUR5XEJGU5EA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.464971024 (613h42m35.336054059s) [noise=true , address=145.79.227.3:4163 , piece_id=WGHY2U4QHCY7OD3ZLLSCYCX2P25KFGAC3LSVELDMZODPRJRWL66Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.578492509 (613h42m35.449575544s) [noise=true , address=77.90.31.15:4728 , piece_id=CKN2Y54QODO7AKQ6YTPZ6J75WOYE7LTJYLXGMZROTRM2VUAGANBA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.620142154 (613h42m35.491225189s) [noise=true , address=145.79.227.49:4355 , piece_id=FSIMCZHQ2P5KECWLYZ5265U6S4236KJDG74DR5EDYJWVDCLDFQGQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.621092664 (613h42m35.492175699s) [noise=true , address=77.90.31.21:4886 , piece_id=UJ7N4QKXEECTDW5HSC5EY63SHVF3IFQ4JOVCEZPLEMXIQ4CEI4JQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:35.702975045 (613h42m35.57405808s) [noise=true , address=77.90.31.16:4752 , piece_id=HSLBOHNLT7LZ74HXGYC6LGK5HMWY7VM35OV7RXUWHA557KFIVNRA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.227756984 (613h42m36.098840029s) [noise=true , address=193.149.164.143:4698 , piece_id=OUZTAUGILXAOW7AAABBWULPA5AYDX6SLXSATNGBBWNIOVA3CHWLA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.392581293 (613h42m36.263664328s) [noise=true , address=145.79.227.30:4787 , piece_id=DBWLIUGFV245WD4IQWTXNFVSDI4QUERXMDMUMNGGSVXLGFDFNIEA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.399008411 (613h42m36.270091456s) [noise=true , address=145.79.227.33:4892 , piece_id=NK44ZMVQOQMSIO6Q2T37Z4WEMFTG5QNUX2VQ6GQLZTCZCH3WBASQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.402448390 (613h42m36.273531425s) [noise=true , address=145.79.227.44:4227 , piece_id=BM7AJBVEI5OYEBHD2GDDQKW6PVYV7MXL66NPSF6PLILCR4HWWPAQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.445018687 (613h42m36.316101732s) [noise=true , address=145.79.227.40:4059 , piece_id=43AKNYVOAWIJZTV3LLEIN3EIZDLLO2SIPRTF6BVG3B7G2GZTQI7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.632012951 (613h42m36.503095986s) [noise=true , address=45.91.116.10:4115 , piece_id=X4WIJR4KUTEAAX2EMSEBBVQCZFEBC5VUB4OZD6HXS627DAUWBH7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697934471 (613h42m36.569017516s) [noise=true , address=77.90.31.14:4694 , piece_id=YFXATUQVOGJXOQSKFCIPW5FMXPZHLWYTAY7NAVNO7EDFDULAHPOA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: <nil>
Dec 31 05:11:36.697957184 (613h42m36.569040219s) [seg_pos={0 0}, upload=537816]: Segment reached optimal threshold of 14 pieces.
Dec 31 05:11:36.698144040 (613h42m36.569227075s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698150162 (613h42m36.569233197s) [noise=true , address=145.79.227.29:4971 , piece_id=VUL2PRKECN5CEXAKMAARSFG3V4RLP25K3PGE75FJGQBRO3W3HTIA, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698260782 (613h42m36.569343817s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2): protocol: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->45.91.116.7:4039: use of closed network connection
Dec 31 05:11:36.698266192 (613h42m36.569349228s) [noise=true , address=45.91.116.7:4039 , piece_id=GVC3YVNFRHJF2MUS5KKX6HDHJWCQTW46BZK344VQBOXLICFAVH3A, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698387063 (613h42m36.569470098s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 31 05:11:36.698394898 (613h42m36.569477933s) [noise=true , address=145.79.227.34:4930 , piece_id=WMX7TFPMXC3Y4SSIJLXL6LZBDJI7D72H7CN7NVJZP53NH5RZ2I7Q, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698402151 (613h42m36.569485186s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42470->64.49.13.169:4709: use of closed network connection
Dec 31 05:11:36.698408293 (613h42m36.569491328s) [noise=true , address=64.49.13.169:4709 , piece_id=PYXDLYSVK6QKTFG56425VJJP5B76TYVJ5YBVKS4UVRL6Z4WZZBRQ, seg_pos={0 0}, upload=537816]: Long tail cancellation due to optimal threshold
Dec 31 05:11:36.698485230 (613h42m36.569568265s) [seg_pos={0 0}, upload=537816]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 05:11:36.698568288 (613h42m36.569651323s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:11:36.698574279 (613h42m36.569657314s) [upload=537816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:11:36.985448011 (613h42m36.856531057s) [upload=537816]: Done uploading segments...
Dec 31 05:11:36.985472999 (613h42m36.856556034s) [upload=537816]: Upload finished. err: <nil>
Duration: 1m0.005274278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [......X.......X.........XX...X.X....X......X.XXX...XX.X.....] duration:
5.68536831s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:12:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 5.685397446s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X...........X..............X.........X.........X...........] duration:
3.550976188s trace:
https://jaeger.dev.storj.io/trace/2b1c6f2c257d6b6b
FAIL (2025-12-31 05:12:21 UTC)
orbiter warning: context deadline exceeded
Duration: 3.55100247s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.0680157s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:12:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.068057789s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration:
8.136482834s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:12:01 UTC)
orbiter warning: context deadline exceeded
Duration: 8.136518563s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.......X.......X.........XX...X.X....X......X.XXX...XX.X....] duration:
5.734206574s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:11:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 5.734226062s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X.......X.......X.........XX...X.X....X......X.XXX...XX.X...] duration:
6.280106075s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:11:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 6.280137144s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration:
20.666076614s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:11:38 UTC)
orbiter warning: context deadline exceeded
Duration: 20.666124956s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.219980885s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:11:23 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.220011824s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
33.887338382s trace:
https://jaeger.dev.storj.io/trace/5943c4171683f71a
FAIL (2025-12-31 05:11:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment
Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments...
Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces.
Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection
Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments...
Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil>
Duration: 33.887368047s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X...........X..............X.........X.........X..........] duration:
4.646187782s trace:
https://jaeger.dev.storj.io/trace/2b1c6f2c257d6b6b
FAIL (2025-12-31 05:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 4.648188842s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X.......X.......X.........XX...X.X....X......X.XXX...XX.X..] duration:
6.092650554s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:11:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 6.092679501s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X...........X..............X.........X.........X.........] duration:
3.662410277s trace:
https://jaeger.dev.storj.io/trace/2b1c6f2c257d6b6b
FAIL (2025-12-31 05:10:54 UTC)
orbiter warning: context deadline exceeded
Duration: 3.662480249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
39.65273248s trace:
https://jaeger.dev.storj.io/trace/5943c4171683f71a
FAIL (2025-12-31 05:10:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece...
Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment
Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments...
Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil>
Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces.
Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection
Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF
Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold
Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments...
Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil>
Duration: 39.652798667s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X...........X..............X.........X.........X........] duration:
3.592112047s trace:
https://jaeger.dev.storj.io/trace/2b1c6f2c257d6b6b
FAIL (2025-12-31 05:10:23 UTC)
orbiter warning: context deadline exceeded
Duration: 3.592138888s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X.......X.......X.........XX...X.X....X......X.XXX...XX.X.] duration:
6.064723726s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:10:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 6.064746881s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.093542604s trace:
https://jaeger.dev.storj.io/trace/3d2998323a09f583
FAIL (2025-12-31 05:10:18 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 5.09357722s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration:
6.148878385s trace:
https://jaeger.dev.storj.io/trace/34f8006271c37a9d
FAIL (2025-12-31 05:10:13 UTC)
orbiter warning: context deadline exceeded
Duration: 6.148906148s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...X.......X.......X.........XX...X.X....X......X.XXX...XX.X] duration:
30.003009863s trace:
https://jaeger.dev.storj.io/trace/43977ae61decf39d
FAIL (2025-12-31 05:10:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA) context deadline exceeded
======LOGS======
c 31 05:09:42.724827552 (613h40m42.595910587s) [upload=537791]: Starting upload
Dec 31 05:09:42.724880843 (613h40m42.595963878s) [upload=537791]: Done starting upload
Dec 31 05:09:42.724976736 (613h40m42.596059781s) [upload=537791]: Uploading segments...
Dec 31 05:09:42.725144285 (613h40m42.59622733s) [upload=537791]: Got next segment. Inline: false
Dec 31 05:09:42.725175394 (613h40m42.596258429s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 05:09:42.725180254 (613h40m42.596263289s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 05:09:42.970199750 (613h40m42.841282785s) [seg_pos={0 0}, upload=537791]: Begin upload segment...
Dec 31 05:09:42.970260857 (613h40m42.841343892s) [seg_pos={0 0}, upload=537791]: Done begin upload segment.
Dec 31 05:09:42.970334096 (613h40m42.841417131s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970344336 (613h40m42.841427371s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.970958455 (613h40m42.8420415s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971361883 (613h40m42.842444928s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.971829564 (613h40m42.842912609s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.972803108 (613h40m42.843886163s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.973468966 (613h40m42.844552001s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.974198044 (613h40m42.845281079s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975443986 (613h40m42.846527031s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.975872942 (613h40m42.846955987s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.977858423 (613h40m42.848941468s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.979571596 (613h40m42.850654641s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.981453690 (613h40m42.852536735s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.982750509 (613h40m42.853833554s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.987765246 (613h40m42.858848292s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:42.988600717 (613h40m42.859683762s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Uploading piece...
Dec 31 05:09:44.575320105 (613h40m44.44640315s) [upload=537791]: Next returned nil segment
Dec 31 05:09:44.575332950 (613h40m44.446415985s) [upload=537791]: Waiting for error group managing segments...
Dec 31 05:09:44.957012781 (613h40m44.828095826s) [noise=true , address=145.79.227.27:4691 , piece_id=DFF6ZIVV7MHQEM6BWPYRSYRQKIBDIQV5BMB6DJWQJA7WUNBDFHGQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.965194630 (613h40m44.836277665s) [noise=true , address=145.79.227.14:4838 , piece_id=OCMFQE47A4FDGYBI5E2QQXWYX2C6JA5HS4A4L3WKLSYMFJEA77VQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967120266 (613h40m44.838203301s) [noise=true , address=145.79.227.39:4014 , piece_id=UUNV7AP3LGMPYKTQFI37PNU6RMJPPFNUIKW7XAJKNL3X764ZROUQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.967963942 (613h40m44.839046987s) [noise=true , address=145.79.227.15:4875 , piece_id=PX7WEPFX6YIN4MH7ZAWUWOA6SEZQZD5H3RQKYPHYSJA4IBNKYBTA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.974318933 (613h40m44.845401978s) [noise=true , address=145.79.227.25:4618 , piece_id=AEVZSM4F7QJVLYAYO2LLVQRMR3OQ3ZXQTVZG7CRXHYAMZZSOHLOQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.975066156 (613h40m44.846149201s) [noise=true , address=145.79.227.6:4257 , piece_id=YU5Q4A4JZSE6QUJFWQZFXDH7WIQME6DSMECWMVOO6OU4FBLGH47Q, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.979453229 (613h40m44.850536264s) [noise=true , address=145.79.227.9:4365 , piece_id=7XJJ5HU2N4SXDVAJSBRVJVWKQ7ZHHRN3G5ICTR7JHW4JOQK2YDWQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:44.987919700 (613h40m44.859002745s) [noise=true , address=145.79.227.47:4266 , piece_id=GIC2VS777MHA3D3SQFH2B64KPRTGFITLOASC5JJUMDTVOWS6L2UQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.267203129 (613h40m45.138286194s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ46CTEOXMHH3XQPV7RSPEJV4EA7NBKKMZDOCB5V5T7LKRKNZIDA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.270454290 (613h40m45.141537325s) [noise=true , address=77.90.31.16:4735 , piece_id=DPDPYDWNTTMZKEI3K4VSAJEB4SSHALOBYMS2MIACSWAZKOAICLHQ, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.337130648 (613h40m45.208213683s) [noise=true , address=45.91.116.10:4084 , piece_id=6GMVCDBJLU3SXJYHFMXT4S3RVPACLDXWM6NRBI3VLL5JSVZWNJ3A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.364729748 (613h40m45.235812793s) [noise=true , address=45.91.116.14:4190 , piece_id=BOMND4WJGR3G26IEF6K7XUQO57BQAL4MPQQL7KGPQQ23HQ5SL32A, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408182256 (613h40m45.279265291s) [noise=true , address=45.91.116.6:4974 , piece_id=YFD6ZLCQSTCRTKZMTWS2SJU5NJT5T63GFYXIE4XF2RPBL2TXUKKA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: <nil>
Dec 31 05:09:45.408203416 (613h40m45.279286461s) [seg_pos={0 0}, upload=537791]: Segment reached optimal threshold of 13 pieces.
Dec 31 05:09:45.408382907 (613h40m45.279465952s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408391775 (613h40m45.27947482s) [noise=true , address=77.90.31.14:4661 , piece_id=X7DG3NF54DHNAWNYOXGMS2XKRWKG5KQDSO3N4SPAGCCBMOT4LXFA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408648103 (613h40m45.279731138s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37676->45.91.116.5:4299: use of closed network connection
Dec 31 05:09:45.408654865 (613h40m45.2797379s) [noise=true , address=45.91.116.5:4299 , piece_id=7TIZZAQSQJY7VUJ222T2X2IKX2VTHTT2SZWMGQ7BM652JHMO5QQA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408763021 (613h40m45.279846056s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 05:09:45.408768822 (613h40m45.279851867s) [noise=true , address=145.79.227.29:4941 , piece_id=7FXL3YYU5E3TYZCQKGWTD5EBMCAE5X6BDWCRLFAD5OYYERV76YNA, seg_pos={0 0}, upload=537791]: Long tail cancellation due to optimal threshold
Dec 31 05:09:45.408786496 (613h40m45.279869531s) [seg_pos={0 0}, upload=537791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 05:09:45.408857111 (613h40m45.279940146s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 05:09:45.408862451 (613h40m45.279945486s) [upload=537791]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 05:09:45.704241388 (613h40m45.575324423s) [upload=537791]: Done uploading segments...
Dec 31 05:09:45.704258300 (613h40m45.575341345s) [upload=537791]: Upload finished. err: <nil>
Duration: 30.003034399s
core
uplink
us-select-1
|